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putting a syiithesized result, and a terminal control sec- 
tion 18 for controlling the atx>ve means makes It 
possil^le to synthesize a plurality of videos at the same 
time and correspond to a dynamic change of transmis- 
sion format information. 




Printed by XercK (UK) Business Services 
2.16.7/3.6 



EP0 905 976A1 



Description 

Technical Retd 

[0001] The present invention relates to audio-video 
transmitter and audio-video receiver, data-processing 
apparatus and method, waveform-dala-transmitting 
method and apparatus and wavefbrm-data-receiving 
method and apparatus, and video-transmitting method 
and appsiratus and video-receiving method and appara- 
tus. 

Background Art 

[0002] There has been an apparatus which satisfies 
the sense of real existence that a counterpart is present 
in front of you and aims at realistic picture communica- 
tion tiy extracting, for exanple, a person's picture out of 
the scenery picture of a space in which you are present 
and superimposing the person's picture, a person's pic- 
ture sent from the counterpart, and the picture of a vir- 
tual space to be displayed commonly with a previously- 
stored counterpart on each other and displaying them 
(Japanese Patent Put^lication No. 4-24914). 
[0003] Particularly, in the case of the prior art, inven- 
tions concerned with acceleration for performing picture 
synthesis and a method for reducing memories are 
made (e.g. Official gazette of Japanese Patent Publica- 
tion No. 5-46592: Picture synthesizer). 
[0004] Though a communication system using picture 
synthesis for synthesizing two-dimensional static pic- 
tures or three-dimensional CG data has been proposed 
by the prior art, specific discussion on a method for real- 
izing a system for simultaneously synthesizing a plural- 
ity of video (picture) and a plurality of audio and 
displaying them hias not been performed from the fol- 
lowing viewpoints. 

[0005] That is, there has been a problem that no spe- 
cific discussion has been performed from the following 

viewpoints: 

(A1) a method for transmitting (communicating and 
broadcasting) and controlling pictures and audio 
urKler the environment in which data and control 
information (information transmitted by a packet dif- 
ferent from tiiat of data to control the processing of 
terminal side) are indeperxlently transmitted by 
using a plurality of logical transmission lines con- 
structed by software on one rea\ transmission Ijne 
or more; 

(A2) a method for dynamically changing header 
information (corresponcfing to data control informa- 
tion of the present invention) to be added to data for 
a picture or audio to be transmitted; 
(A3) a method for dynamically changing header 
information (corresponding to transmission control 
information of the present invention) to be added for 
transmission; 



(A4) a method for transmitting information by 
dynamically multiplexing and separating a plurality 
of logical transmission lines; 
(A5) a method for transmitting pictures and audio 
5 considering the read and rise periods of program or 
data; and 

(A6) a metfKxJ for transmitting pfotures and audio 
considering zapping. * 

10 [0006] However, the method for changing encoding 
systems and a method of discussing data in frames in 
accordance with the frame type of a picture have been 
proposed so far as a method for dynamically adjt^ng 
the amount of data to be transmitted to a network (H. 

15 Jinzenji and T. Tajiri. A study of distributive-adaptive- 
type VOD system. D-81, System Society of Institute of 
Electronics. Information and Communication Engineers 
(lEICE) (1995)). 

[0007] A dynamic throughput scalable algoritfim capa- 
20 ble of providing a high-quality video under a restricted 
processing time is proposed as a metiiod for adjusting 
V throughput at the encoder side (T. Osako. Y. YEtjima, H. 
Kodera. H. Watanabe. K. Shimanrura: Encoding of soft- 
ware video using a dynamic throughput scalatrie algo- 
us ritiim, Thesis Journal of lEICE. D-2, Vol. 80-D-2. No. 2. 
pp. 444-458 (1997)). 

[0008] Moreover, there is an MPEGiyMPEG2 system 
as an example of realizing synchronous reproductfon of 
video and audio. 

30 

(B1) The conventional method for discussing a pic- 
ture correspondingly to the frame type of the video 
has a problem that it is difficult to prepdnderantiy 
reproduce an inportant scene cut synchronously 
35 with audio by handling a plurality of video streams 
or a plurality of audio streams and reflecting the 
intention of an editor because the grading of the 
information which can be handled is in a single 
stream. 

40 (B2) Moreover, it must be possible that a decoder 
decodes every supplied bit stream because it is a 
prerequisite tiiat MPEG1/MPEGi2 is realized by 
hardware. Therefore, it is a problem how to corre- 
spond to the case of exceeding the tivoughput of 
45 the decoder. 

[0009] Moreover, to transmit video, there have been 
some systems including a system such as H. 261 (ITU- 
T Recommendation H. 261 -Video codec for audio-vis- 
50 uai services at px 64) and they have been mounted by 
hardware. Therefore, the case has not occurred that 
decoding is not completed within a designated time 
because of considering the upper limit of a necessary 
performance when designing hardware. 
55 [0010] The above-designated time denotes a time 
required to transmit a bit stream obtained by coding a 
sheet of video. If decoding is not completed within the 
time, an extra time becomes a delay. If the delay is accu- 



25 



30 



35 



40 



45 



50 



2 



3 



EP0905 976A1 



4 



mutated, the delay from the transmitting side to the 
receiving side increases and the system cannot be used 
as a video telephone. This state must be avoided. 
[0011] Moreover, when decoding cannot be com- 
pleted within a designated tme because a comriiunica-' s 
tion counterpart generates an out-of-spec bit stream, a 
problem occurs that a video cannot be transmitted. 
[001 2) The above problem occurs not only for a video 
but also for audio data. 

[001 3] However, in recent years, because the network io 
environment formed by personal computers (PCs) has 
been arranged as the result of spread of internet and 
ISDN, the transmission rate has been improved and it 
has been possible to transmit a video by using PCs and 
a network. Moreover, requests for transmission of video is 
by users have been rapidly increased. Furthermore, a 
video can be completely decoded by software because 
CPU performances have been improved. 
[0014] However, because the same software can be 
executed by personal computers different in structure 20 
such as a CPU, bus width, or accelerator, it is difficult to 
previously consider the upper limit of a necessary per- 
formance and ttierefore, a problem occurs that a picture 
cannot be decoded within a designated time. 
[001 5] Moreover, when coded data for a video having 25 
a length exceeding the throughput of a receiver is trans- 
mitted, coding cannot be completed within a designated 
time. 

Problem (C1): Decreasing a delay by decoding a 30 
picture within a designated time. 

When inputting a video as the waveform data of 
claim CI of the present invention or outputting a 
video as the waveform data of claim C7 of the 
present invention as means for solving the prot)lem 35 
1. a problem may be left that the substantial work- 
ing efficiency of a transmission line is lowered 
because a part of a transmitted bit stream is not 
used. Moreover, there are some coding systems 
that generate a present decoded video in accord- 40 
ance with a last decoded picture (e.g. P picture). 
However, because the last decoded picture is not 
completely restored by the means for solving the 
problem 1 , there is a problem that deterioration of 
the picture quality influentially increases as time 4S 
passes. 

Problem (C2): in the case of the means for solving 
the problem 1, the substantial working efficiency of 
a transmission line is lowered. Moreover, picture- 
quality deterioration is spread. so 

Furthermore, in the case of mounting by soft- 
ware, the frame rate of a picture is determined by 
the time required for one-time coding. Therefore, 
when the frame rate designated by a user exceeds 
the throughput of a computer, it is impossible to cor- ss 
respond to the designation. 
Problem (C3): When the frame rate designated by a 
user exceeds the throughput of a computer, it is 



impossible to con^espond to the designation. 
Disclosure of the Invention 

[001 6] When considering the problems (A1 ) to (A6) of 
the first prior art, it is an object of the present invention 
to provide an audio-video transmitter and audio-video 
receiver and data-processing apparatus and method in 
order to solve at least any one of thla problems. 
[001 7] Moreover, when considering the problems (B1 ) 
and (B2) of the second prior art, it is another object of 
the present invention to provide data-processing appa- 
ratus and method in order to solve at least one of the 
problems. 

[0018] Furthermore, when considering the prot)tems 
(CI) to (C3) of the last prior art, it Is still another object 
of the present invention to provide wavefprm-data- 
receiving method and apparatus and waveform-data- 
transmftting method arid apparatus, and video-transmit- 
ting method and apparatus and vkJeo-receiving method 
and apparatus in order to solve at least one of the prot> 
lems. 

[0019] The present invention according to daim 1 is 
an audio-vkieo transnriitting apparatos comprising 

transmitting means for transmitting the content con- 
cerned with a transmitting method and/or the struc- 
ture of data to be transmitted or an klentifier 
showing the conterit as ti^ansmissfon format infor- 
mation through a transmission line same as that of 
the data to be transmitted or a transmission line dif- 
ferent from the data transmission line: wherein 
said data to be transmitted is vkJeo data andJor 
audfodata. 

[0020] The present invention according to daim 2 is 
the audio-video ta'ansmitting apparatus according to 
daim 1 . wherein said transmission format information is 
included in at least one of data control information 
added to said data to contrd said data, transmission 
control information added to said data to transmit said 
data, and information for controlling the processing of 
the terminal side. 

[0021] The present invention according to claim 3 is 
ttie audio-video f ansmitting apparatus according to 
daim 2, wherein at least one of said data control infor- 
mation, transmission control information, and informa- 
tion for controlling the processing of said terminal side is 
dynamically changed. 

[0022] The present invention according to claim 4 is 
tiie audio-video transmitting sqsparatus according to 
daim 3, wherein said data is divided into a plurality of 
packets, and said data control inforrnation or said trar^- 
mission contrd information is added not only to the 
head packet of said divided packets but also to a middle 
packet of them. 

[0023] The present invention according to daim 5 is 
tiie audio-video transmitting apparatus according to 
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claim 1. wherein an identifier showing whether to use 
timing information concerned with said datii as infomna- 
tion showing the r^roducing time of said data is 
Included in said transmission format information. 
[0024] The present invention according to daim 6 is 
the audio-video transmitting apparatus according to 
claim 1 , wherein said transmission format information is 
the structural information of said data and a signal 
which is output from a receiving apparatus receiving the 
transmitted structural information of said data and which 
can be received is confirmed and thereafter, said trans- 
mitting means transmits corresponding data to said 
receiA^ng apparatus. 

[0025] The present invention according to daim 7 is 
the audio-video transmitting apparatus according to 
claim 1, wherein said transmission format information 
indude (1) an Identifier for Identifying a program or data 
to be used by a receiving apparatus later and (2) at least 
one of a flag, counter, and timer as Information for 
knowing the point of time in which said program or data 
is used or the term of validity for using said program or 
data. 

[0026] The present invention according to daim 8 is 
the audio-video transmitting apparatus according to 
daim 7. wherein said point of time in which said pro- 
gram or data is used Is transmitted as transmissiori con- 
trol information 1:^ using a tfEUismission serial nun1t>er 
for identifying a transmission sequence or as informa- 
tion to be transmitted by a packet different from that of 
data to control terminal -side processing. 
[0027] The present inventksn according to daim 9 is 
the audio-video transmitting apparatus according to 
claim 2 or 3. wherein storing means for storing a plural- 
ity of contents concerned witii said transmitting method 
and/or said structure of data to be transmitted and a plu- 
rality of its identifiers are induded. and said identifier is 
Induded in at least one of said data control information, 
transmission control information, and irrformatlon for 
controlling terminal-side processing as said transmis- 
sion format information. 

[0028] The present invention according to daim 10 is 
the audio-video transmitting apparatus according to 
claim 2 or 3, wherein storing means for storing a plural- 
ity of contents concerned witii said transmitting method 
and/or said structure of data to be transmitted are 
included, and said contents are included in at least one 
of said data control infbrnnation. transmission control 
information, and information for controlling terminal-side 
processing as said transmission format information. 
[0029] The present invention according to daim 1 1 is 
the audio-video transrhitting apparatus aoc-0!tfing to 
claim 1, 2. or 3, wherein a def^uft identifier showing 
whether to change the contents concerned with said 
transmitting method and/or structure of data to be trans- 
mitted is added. 

[0030] The present invention according to daim 12 is 
the audio-video transmitting apparatus according to 
claim 9, 10. or 1 1 , wherein said Identifier or sakJ deteutt 



identifier is added to a predetermined fixed-length 
region of information to be transmitted or said predeter- 
mined position. 

[0031] The present invention according to daim 13 is 
5 an audk>-video receiving apparatus comprising: receiv- 
ing means for receiving said transmission format infor- 
mation transmitted from the audio-video transmitting 
apparatus of any one of daims 1 to 12; and transmitted- 
information interpreting means for interpreting said 
10 received transmission-format information. 

[0032] The present invention according to daim 1 4 is 
the audio-video receiving apparatus according to daim 
13. wherein storing means for storing a plurality of con- 
tents concerned with said transmitting method and/or 
t5 said structure of data to be transmitted and a plurality of 
its identifiers are included, and the contents stored in 
said storing means are used to interpret said transmis- 
sion format information. 

[0033] The present Invention according to daim 1 5 is 
20 an audiovideo transmitting apparatus comprising: Infor- 
mation multiplexing means for controlling start and end 
of multiplexing the information for a plurality of logical 
transmission lines for transmitting data and/or control 
information is Induded; wherein, not only said data 
55 and/or control information multiplexed by said informa- 
tion nnultiplexing means but also control contents con- 
cerned with start and end of said multiplexing by said 
information multiplexing means are transmitted as mul- 
tiplexing control information, and said data Indudes 
3o video data and/or audio data. 

[0034] The present Invention according to claim 1 6 is 
the audio-video transmitting apparatus according to 
daim 15. wherein it is possible to select whetiier to 
transmit said multiplexing control Information by arrang- 
35 ing said information without multplexihg ft before said 
data and/or control information or transmit said multi- 
plexing corrtrd information through a transmission line 
different from the transmission line for transmitting said 
data and/or control information. 
40 [p035] Thepresent invention according to daim 17 is 
an audio-video receiving apparatus comprising: receiv- 
ing means for receiving said multiplexing contrd infor- 
mation transmitted from the audio-video transmitting 
apparatus of claim 15 and said multiplexed data aixl/or 
45 control information; arxi separating liieans for separat- 
ing said multiplexed data and/or control infbrnnation in 
accordance with said multiplexing control information. 
[0036] The present invention according to daim 18 is 
an audio-video receiving apparatus corTf)rising: main 
50 looking-listening means for looking at and listening to a 



iCOKing-listei lii ly 



means for cyclically detecting ttie state of a broadcast 
program other tiian tiie broadcast program looked and 
listened through said main looWng-listening means; 
55 wherein said detection is performed so that a program 
and/or data necessary when said broadcast program 
looked and listened through said miain looking-listening 
means Is switched to other broadcast program can be 
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smoothly processed; and 

said data includes video data and/or audio data. 
[0037] The present invention according to claim 19 is 
the audio-video transmitting apparatus according to 
claim 1, wherein priority values can be changed in s 
accordance with the situation by transmitting the offset 
value of information showing the priority for processing 
of siaid data. 

[0038] The present invention according to claim 20 is 
an audio>video receiving apparatus comprising: receiv- io 
ing means for receiving encoded information to which 
the information concerned with the priority for process^ 
ing under an overload state is previously added; and pri- 
ority deciding means for deciding a threshold serving as 
a aiterion for selecting whether to process an object in is 
said information received by said receiving means; 
wherein 

the timing for oulputting said received informa- 
tion is compared with the elapsed time after start of 
processing or the timing for decoding said received 20 
information is compared with the elapsed time after 
start of processing to change said threshold in accord- 
ance with the comparison result, and video data and/or 
audio data are or is included as said encoding object. 
[0039] The present invention according to claim 21 is 25 
the audio-video receiving apparatus according to claim 
20, wherein retransmission-request-priority deciding 
means for deciding a threshold serving as a criterion for 
selecting whether to request retransmission of some of 
said information not received because it is lost under 30 
transmission when it is necessary to retransmit said 
information is included, and 

said decided threshold is decided in accordance 
with at least one of the priority controlled by said priority 
deciding means, retransmission frequency, lost factor of 35 
information, insertion interval between in-frame- 
encoded frames, and grading of priority. 
[0040] The present invention according to claim 22 is 
an audio-video transmitting apparatus comprising: 
retransmission-priority deciding means for deciding a 40 
threshold serving as a criterion for selecting whetiier to 
request retransmission of some of said information not 
received because it Is lost under transmissfon when 
retransmission of said unreceived information is 
requested is included, wherein said decided threshold is 45 
decided in accordance with at least one of the priority 
controlled by the priority deciding means of said audio- 
video receiving appairatus of claim 20. retransmission 
frequency, lost factor of Information, insertion interval 
between in-frame-encoded frames, and grading of prior- so 

[0041] The present invention according to claim 23 is 
an audio-video tiransmitting apparatus for transmitting 
said encoded information by using the priority added to 
said encoded information and thereby thinning it when ss 
(1) an actual transfer rate exceeds the target transfer 
rate of Information for a video or audio or (2) it Is 
decided that writing of said encoded information Into a 



transmitting buffer is delayed as tiie result of conparing 
ttie elapsed time after start of transmission with a period 
to be decoded or output added to said encoded informa- 
tion. 

[0042] The present invention according to claim 25 is 
a data processing apparatus corrprising: receiving 
means for receiving a data series irKluding (1) time- 
series data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the processing 
between said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said time- 
series data value to show the processing priority 
between divided data values; and data processing 
means for performing Processing by using said inter- 
time-series-data priority and said In-time-series-data 
priority together when pluralities of said time-series- 
data values are. simultaneously present 
[0043] The present invention according to claim 27 is 
a data processing apparatus comprising: receiving 
means for receiving a data series including (1) time- 
series data for audio or video, (2) ari inter-time-series- 
data priority showing the priority of the processing 
between said time-series-data values, and (3) a plurality 
of In-time-series-data priorities for dividing said time- 
series data value to show the processing priority 
between divided data values; and data processing 
means for distributing throughput to each of said time- 
series-data values in accordance with said inter-time- 
series-data priority and moreover, adaptively deteriorat- 
ing the processing quality of the divided data in said 
time-series data In accordance with said in-time-series- 
data priority so that each of said time-series-data values 
is kept witiiin said distributed throughput. 
[0044] The present invention according to claim 29 is 
a data processing apparatus characterized by. when an 
in-time-series-data priority for a video is added every 
frame of said video and said video for each frame is 
divided into a plurality of packets, adding said in-time- 
series-data priority only to the header portion of a 
packet for transmitting the head portion of a frame of 
said video accessible as independent information. 
[0045] The present invention according to claim 31 is 
the data processing apparatus according to any one of 
daims 25. 27, and 29, wherein said in-time-series-data 
priority is described in the header of a packet to perform 
priority processing. 

[0046] The present Invention according to claim 33 is 
the data processing apparatus according to any one of 
claims 25, 27, and 29, wherein the range of a value 
capak)le of expressing said in-time-series-data priority is 
made variable to perform priority processing. 
[0047] The present invention according to claim 34 is 
a data processing method comprising the steps of: 
Inputting a data series Including time-series data for 
audio or video and an inter-time-series-data priority 
showing the processing priority between said time- 
series data values; and 

processing priorities by using said inter-time- 
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series<iata priority as the value of a relative or absolute 
priority. 

[0048] The present invention according to daim 36 is 
a data processing method comprising the steps of: clas- 
sifying time-series data values for audio or video: input- s 
ting a data series including said time-series data and a 
plurality of in-time-series-data priorities showing the 
processing priority between said classified data values; 
and processing priorities by using said in-time-series- 
data priority as the value of a relative or absolute prior- 
ity. 

[0049] Moreover, to solve the problem (CI), the 
present Invention is characterized by: 



[0050] Moreover, to solve the problem (C2). the 
present invention is characterized by: 



[0051] Furthermore, to solve the problem (C3), the 
present invention is characterized by: 

40 

(d) estimating the execution time of each group in 
accordance with the processing time required to 
encode a video and each execution frequency out- 
put by coimting means; and 

(e) estimating the processing time required to 4s 
encode a video by using the above execution time . 
and computing the execution frequency of eiach 
group in which the processing time does not 
exceed a time usable to process one sheet of pic- 
ture determined by a frame rate given as the desig- so 
nation of a user in acccruance with the waveform- 
data-transmitting method of daim 67. 

[0052] The present invention has the above structure 
to obtain the execution frequency of indispensable ss 
processing and that of dispensable processing, transmit 
the execution frequencies to the receiving side, and 
estimate the time reqidred for each processing in 



accordance with the execution frequencies and the 
decoding time. 

[0053] By redudng each execution frequency of dis- 
pensable processing so . that the time required for 
decoding becomes shorter than a designated time in 
accordance with the estimated time of each processing, 
it is possible to control the decoding time to the desig- 
nated time or shorter and keep a delay Ismail. 
[0054] Claims 67 and 73 are mainly fisted as the 
inventions for solving the protrfem (CI ), 
[0055] Moreover, it ^ possible to set the decoding exe- 
cution time to a value equal to or less than a designated 
time by trguismitting the execution time of Indispensable 
processing and that of dispensable processing esti- 
mated by the receiving side to the transmitting side and 
determining each execution frequerK^y at the transnrvt- 
ting side in accordance with each execution time. 
[0056] aaims 75 and 77 are mainly fisted as tiie 
inventions for solving the problem (C2). 
[0057] Moreover, it is possible to set the encoding esti- 
mation time to a value equal to or less than a user des- 
ignated time by estimating the execution time of 
indispensable processing and that of dispensable 
processing and determining each execution frequency 
in accordance with each execution time and the user 
designated time determined by a frame rate designated 
by a user. 

[0058] Claim 79 is mainly listed as the Invention for 
solving the problem (C3). 

Brief Description of the Drawings 

[0059] 

Rgure 1 is a schematic block diagram of the aucfio- 
video transceiver of an embodiment of the present 

invention; 

Rgure 2 is an illustration showing a reception con- 
trol section and a separating section; 
Rgure 3 is an illustration shoving a method for 
transmitting and controlling video and aucfio by 
L^ing a plurality of logical transmission lines; 
Rgure 4 is an illustration showing a method for 
dynamically changing header information added to 
tiie data for a video or audio to be transmitted; 
Rgures 5(a) and 5(b) are illustrations showing a 
method for adding AL information; 
Rgures 6(a) to 6(cl) are illustrations showing exam- 
ples of a mettKXj for adding AL information; 
Rgure 7 is an illustration showing a metfxxl for 
transmitting information by dynamically multiplexing 
and separating a plurality of logical transmissk>n 
lines; 

Rgure 8 is an illustration showing a procedure for 
transmitting a broadcasting program; 
Rgure 9(a) is an illustration showing a method for 
tiansmitting a video or audio considering the read 
and rise time of program or data when tiie program 



iriputting, for example, a video as waveform data In is 
accordance with the waveform-data-transmitting 
method of daim 63; or 

outputting, for exanple, a video as waveform data 
in accordance with the wavefbrm-data-receiving 
method of daim 69. 20 



(d) outputting the execution time of each group 25 
obtained through estimation in accordance with the 
waveform-data-receiving method of daim 69;or 

(d) inputting a data string constituted with the exe- 
cution time of each group; and 

(e) computing the execution frequency of each 30 
group for conpleting decoding within a time 
required to transmit a code length determined by 
the designation of a rate controller or the like in 
accordance with each execution time of the receiv- 
ing means in accordance witti the wave-data-trans- 3s 
mitting metiiod of daim 63. 
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or data is present at a receiving terminal; 

Rgure 9(b) is an illustration showing a method for 

transmitting a video or audio considering the read 

and rise time of program or data when the program 

or data is transmitted: 5 

Figure 10(a) is an illustration showing a method for 

corresponding to zapping; 

Figure 10(b) is an illustration showing a method for 

correspording to zapping; 

Figure 11(a) is an illustration showing a specific io 
example of the protocol to be actually transferred 
between terminals; 

Figure 11(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; is 
Rgure 12 is an illustration showing a specific exam- 
ple of tiie protocol to be actually transferred 
between terminals; 

Figure 13(a) is an illiistration showing a specific 
example of the protocol to be actually transferred 20. 
between terminals; 

Figure 13(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 

Rgure 13(c) is an illustration showing a specific 2S 
example of the protocol to be actually transferred 
between terminals; 

Figure 14 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 30 
Figure 15 Is an illustration showing a specific exam- 
ple of tiie protocol to be actually transferred 
t>etween terminals; 

Figure 16(a) is an illustration showing a specific 
example of the protocol to be actually transferred 35 
between terminals; 

Figure 16(b) is an illustratioii showing a specific 
example of the protocol to be actually transfen'ed 
between terminals: 

Rgure 17 is an illustration showing a specific exam- 40 
pie of the protocol to be actually transferred 
between terminals; 

Figure 18 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 45 
Figure 19(a) is an illustration showing a specific 
example of the protocol to be actually transfen'ed 
between terminals: 

Figure 19(b) is an illustration showing a specific 
example of the protocol to be actually transferred so 
k>etween terminals; 

Figures 20(a) to 20(c) are block diagrams of dem- 
onstration systems of CGD of the present invention; 
Figure 21 is an illustration showing a method for 
adding a priority under overload at an encoder; ss 
Figure 22 is an illustration describing a metiiod for 
deciding a priority at a receiving terminal under 
overload; 



Rgure 23 is an illustration showing temporal 
change of priorities; 

Figure 24 is an illustration showing stream priority 
and object priority: 

Rgure 25 is a schematic block diagram of a video 
encoder and a video decoder of an embodiment of 
the present invention; 

Rgure 26 is a schematic block diagram of an audfo 
encoder and an audio decoder of an embodiment of 
the present invention; 

Rgures 27(a) and 27(b) are illustrations showirig a 
priority adding section and a priority deciding sec- 
tion for controlling the priority of processing under 
overload; 

Rgures 28(a) to 28(c) are illustrations showing the 
grading for adding a priority; 
Rgure 29 is an illustration showing a method for 
assigning a priority to multi-resolution video data; 
Rgure 30 is an illustration showing a metiiod for 
constituting a communication payload; 
Figure 31 is an illustration showing a method for 
making data correspond to a commuhicatfon pay- 
load; 

Rgure 32 is an illustration showing the relation 
k>etween object priority, stream priority, and commu- 
nication packet priority; 

Rgure 33 is a block diagram of a transmitter of the 
first emtxxiiment of the present invention; 
Rgure 34 is an illustration of the first embodiment; 
Figure 35 is a block diagram of the receiver of the 
third emk)odiment of the present invention; 
Rgure 36 is a block diagram of the receiver of the 
f iftii embodiment of the present invention; 
Rgure 37 is an illusti-ation of the fifth embodiment; 
Figure 38 is a block diagram of the transmitter of the 
sixtii embodiment of the present invention; 
Rgure 39 is a tslock diagram of the trsuismitter of the 
eighth embodiment of the present invention; 
Rgure 40 is a flowchart of the transmission method 
of the second emtxxiiment of the present invention; 
Rgure 41 is a flowchart of tiie reception method of 
the fourth embodiment of the present invention; 
Rgure 42 is a flowchart of the ti^ansmission method 
of tiie seventh embodiment of the present inven- 
tion; 

Rgure 43 is a f fowchart of the transmission method 
of the ninth emtxxiiment of the present invention; 
Rgure 44 is a block diagram showing an audio- 
video transmitter of the present invention; 
Rgure 45 is a t>lock diagram showing an audio- 
video receiver of the present invention; 
Rgure 46 is an illustration for explaining priority 
adding means for adding a priority to a video and 
audio of an audio-video transmitter of the present 
invention; and 

Rgure 47 is an illustration for explaining priority 
deciding means for deciding whether to perform 
decoding by interpreting the priority added to a 
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video and audio of an audio-video receiver of the 
present invention. 

(Description of Syn^ls) 
[0060] 

1 1 Reception control section 

12 Separating section 

13 Transrratting section 

14 Video extending section (Picture extending sec- 
tion) 

15 Video-extension control section (Picture-exten- 
sion control section) 

16 Video synthesizing section (Picture synttiesizing 
section) 

17 Output section 

18 Terminal control section 

401 1 Transmission control section 

4012 Video encoding section (Picture encoding 
section) 

4013 Reception control section 

4014 Video decoding section (Picture decoding 
section) 

4015 Video synthesizing section (Picture synthesiz- 
ing section) 

4016 Output section 

4101 Video encoder (Picture encoder) 

4102 Video decoder (Picture decoder) 

301 Receiving means 

302 Estimating means 

303 Video decoder (i.e. Dynamicisicture or Moving 
picture d^xxJer) 

304 Frequency reducing means 

306 Output terminal 

307 Input ternvnal 

3031 Variable decoding means 

3032 Inverse orthogonal transforming means 

3033 Switching unit 

3034 Movement compensating means 

3035 Execution-time measuring means 

Best Mode for Carrying (Dut the Invention 

[0061] Emtxx£ments of the present invention are 
described below by referring to tiie accompanying draw- 
ings. 

[0062] The entxxjiments described below mainly 
solve any one of the above problems (A1) to (A6). 
[0063] A "picture (or video)" used for tiie present 
invieritfon includes a sfatic-picture and a mcving-picture. 
Moreover, a purposed picture can be a two-dimensional 
picture like computer graphics (CG) or three-dimen- 
sional picture data constituted with a wire-frame model. 
[0064] Rgure 1 is a schematic block diagram of the 
audio-video t^nsceiver of an embodiment of the 
present invention. 

[0065] In Rgure 1. a reception control section 1 1 for 



receiving information and a transmitting section 13 for 
transmitting information are information transnvtting 
means such as a coaxial cable. CATV. LAN. and 
modem. Communication environment can be the envi- 

5 ronment in which a plurality of logical transmission lines 
can be used without considering multiplexing means 
such as intemet or the environment in which multiplex- 
ing means must be considered such as analog tele- 
phone or satellite broadcast 

10 [0066] Moreover, a system for bicGrectionally transfer- 
ring video and audio between terminals such as a pic- 
ture telephone or teleconference system or a system for 
broadcasting broadcast-type video and audio through 
satellite broadcast. CATV, or internet are listed as termi- 

15 nal connection systems. The present invention takes 
such terminal connection systems into consideration. 
[0067] A separating section 12 shown in Rgure 1 is 
means for analyzing received information and separat- 
ing data from control information. Specifically, the sec- 

20 tion 12 is means for decomposing tiie header 
information for transmission added to data and data or 
decomposing the header for data control added to tfie 
data and the contents of the data. A ^cture extending 
section 14 is means for extending a received vkleo. For 

25 example, a vkfeo to be extended can be the com- 
pressed picture of a standardized moving(dynamic) or 
static picture such as H.261 . H,263. MPEG1/2, or JPEG 
or not. 

[0068] The picture-extension control section 1 5 shown 

30 in Rgure 1 is means for monitoring tiie extended state 
of a video. For example, by moiitoring the extended 
state of a picture, it is possible to empty-read a receiving 
buffer without extending the picture when the receiving 
buffer almost causes overt fow and restart the extension 

35 of ttie pfoture after the picture is ready for extensfon. 
[0069] Moreover, in Rgure 1, a picture synthesizing 
section 16 is means for synthesizing an extended pc- 
ture. A picture synthesizing method can be defined by 
describing a picture and its stixictural information (dis- 

40 play position and display time (moreover, a display 
period can be included)), a method for grouping pic- 
tures, a picture display layer (depth), an object ID 
(SSRC to be described later), and tiie relation between 
atfa-ibutes of tiiem with a script lariguage such as JAVA. 

45 VRML, or MHEQ. The script describing the synttiesizing 
method is input or output through a network or a local 
memory. 

[0070] Moreover, an output section 17 is a display or 
printer for outputting a picture syntiiesized result. A tor- 
so riilnal control section 18 Is means fbr controlling each 



extending an audio instead of a picture (it is possible to 
constitute the structure by changing a picture extending 
section to an audio extending section, a picture exten- 
55 sion control section to an audio extension control sec- 
tion, and a picture synthesizing section to an audio 
synttiesizing section) or a structure for extending a pic- 
ture and an audio and synthesizing and displaying tiiem 
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while keeping temporal synchronization. 
[0071 ] Furthermore, it is possible to transmit a picture 
and an audio by using a picture compressing section for 
conrpressing a picture, a picture compression control 
section for controlling the picture compressing section, s 
an audio compressing section for compressing an 
audio, and an audio compression control section for 
controlling the audio compressing section. 
[0072] Figure 2 is an illustredion showing a receptfon 
control, section and a separating section. io 
[0073] By constituting the reception control section 1 1 
shown in Figure 1 with a data receiving section 101 for 
receiving data and a control information receiving sec- 
tion 102 for receiving the control information for control- 
ling data and the separating section 12 with a is 
transmission format storing section 103 for storing a 
transmission structure (to be described later in detail) 
for interpreting transmission contents and a transmis- 
sfon information interpreting section 1 04 for interpreting 
transmission contents in accordance with the transmis- 20 
sion structure stored in the transmission format storing 
section 103. it is possitsle to independently receive data 
and control information. Therefore, for example, it is 
easy to delete or move a received video or audio while 
receiving it. 2s 
[0074] As described above, it Is possible for the com- 
munication environment purposed by the reception con- 
trol section 11 to use a communication environment 
(internet profile) in which a plurality of logical transmis- 
sfon lines can be used without considering multiplexing 30 
means like internet or a communication environment 
(Raw profile) in whidi multiplexing means must be con- 
sidered like analog telephone or satellite k)roadcast. 
However, a user premises a communication environ- 
ment in which a plurality of logical transmission lines 35 
(logical channels) are prepared (for example, In the 
case of a communication environment in which TCP/IP 
can t>e used, the expression referred to as "communica- 
tion porT is generally used). 

[0075] Moreover, as shown in Rgure 2. it is assumed 40 
that the reception control section 11 receives one type 
of data transmission line or more and one type of control 
logical transmission line for controlling data to k>e trans- 
mitted or more. It is also possible to prepare a plurality 
of transmission lines for transmitting data and only one 45 
transmission line for controlling data. Moreover, it Is 
possible to prepare a transmission line for controlling 
data every data transmission like the RTP/RTCP also 
used for H.323. Furthermore, when considering the 
broadcast using UDP, it is possible to use a communica- so 
tion system using a single communtoation port (multi- 
cast address). 

[0076] Rgure 3 is an illustration for explaining a 
method for transmitting and controlling video and audio 
by using a plurality of logical transmission lines. The ss 
data to be transmitted is referred to as ES (Elementary 
Stream). whk:h can be picture information for one frame 
or picture information in GOBs or macroblocks smaller 



than one frame In the case of a picture. 
[0077] In the case of an audio, it is possit)le to use a 
fixed length decided by a user. Moreover, ttie data-con- 
trol header information added to the data to be transmit- 
ted is referred to as AL (Adaptation Layer information). 
The information showing whether it is a start position 
capable of processing data, information showing data- 
reproducing time, and information showing the priority 
of data processing are listed as the AL information. Data 
control information of the present invention corresponds 
to the AL Information. Moreover, it is not always neces- 
sary for the ES and AL used for the present invention to 
coincide with the contents defined MPEGl/2. 
[0078] The information showing whether it is a start 
position capable of processing data specifically includes 
too types of information. First one is a flag for random 
access, that is. tiie information showing tiiat it can be 
individually read and reproduced independently of pre- 
ceding or following data such as intra-frame (I picture) in 
tiie case of a picture; Second one is the information 
capable of defining an access flag as a flag for showing 
tiiat it can t>e individually read, that is, tiie information 
showing that it is the head of pictures in GOBs or mac- 
roblocks in the case of a picture. Therefore, absence of 
an access flag shows the middle of data. Both random 
access flag and access flag are not always necessary 
as the information showing that it is a start position 
capable of processing data. 

[0079] There is a case in which no prot^em occurs 
even if botti the flags are not added in the case of the 
real time communication such as a teleconference sys- 
tem. However, to simply perform edition, a random 
access flag is necessary It Is edso possible to decide 
whether a flag is necessary or which flag is necessary 
through a communication channel t>efore transferring 
data. 

[0080] The information indicating a data reproducing 
time shows the information for time synchronization 
when a picture and an audio are reproduced, which Is 
referred to as PTS (Presentation Time Stamp) in the 
case of MEPGl/2. Because time synchronisation is not 
normally considered in the case of the real time commu- 
nication such as a teleconference system, the informa- 
tion representing a reproducing time is not always 
necessary. The time inten/al between encoded frames 
may be necessary information. 
[0081] By making the receiving side adjust a time 
interval, it is possible to prevent a large fluctoation of 
frame intervals. However, by making the receiving side 
adjust the reproducing interval, a delay may occur. 
Therefore, it may be decided that the time information 
showing the frame interval between encoded frames is 
unnecessary. 

[0082] To deckle whether the information showing a 
data reproducing time represents a PTS or frame inter- 
val, it is also possible to decide tiiat the data reproduc- 
ing time is not added to data before transmitting the 
data and communicate the decision to a receiving termi- 
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nal through the communication channel and transmit 
the data together with decided data control information. 
[0083] When the information showing the priority for 
processing data cannot be processed or transmitted 
due to the load of a receiving terminal or that of a net- 
worK it is possible to reduce the load of the receiving 
terminal or network by stopping the processing or trans- 
mission of the data. 

[0084] The receiving termirial is able to process the 
data with the picture-extension control section 15 and 
the network is aWe to process the data with a relay ter- 
minal or router. The priority can be expressed by a 
numerical value or a flag. Moreover, by transmitting the 
offset value of the information showing the data- 
processing priority as control information or data control 
information (AL information) together with data and add- 
ing the offset value to the priority previously assigned to 
a video or audio in the case of a sudden fluctuation of 
the load of a receiving termineU or network, it is possil)le 
to set a dynarrvc priority corresponding to the operation 
state of a syst^. 

[0085] Furthermore, by transmitting the information for 
identifying presence/absence of scramble, jxes- 
ence/absence of cop/rigfit, and original or copy as con- 
trol information together with a data identifier (SSRC) 
separately from data as control information, it is simpli- 
fied to cancel the scramble at a relay node. 
[0086] Moreover, the Information showing the data 
processing priority can be added every stream consti- 
tuted with the aggregation of frames of a pluriality of pic- 
tures or audios or every frame of video or audio. 
[0087] Priority adding means for deciding the 
encoded-information processing priority under overload 
in accordance with the predetermined rules by the 
encocfing method such as H.263 or Q.723 and making 
the encoded information correspond to the decided pri- 
ority is provkJed for a transmitting terminal unit (see Rg- 
ure 46). 

[0088] Rgure 46 is an illustration for explaining priority 
adding means 5201 for adding a priority to a picture and 
an audio. 

[0089] That is. as shown in Rgure 46. a priority is 
added to encoded-video data (to be processed by video 
encoding means 5202) and encoded-audio data (to be 
processed by audio encocfing means 5203) in accord- 
ance with predetermined rules. The rules for aclding pri- 
orities are stored in priority adding rules 5204. The rules 
include rules for adding a priority higher tiian that of a P- 
f rame (inter-frame encoded pfoture frame) to an t-frame 
(intra-frame encoded picture frame) and rules for ackling 
a pncrity Icrwer uian that of an audio to a picture. More- 
over, it is possible to change the rules in accordance 
with the designation of a user. 

[0090] Priority^dding objects are scene changes in 
the case of a picture or an audio block and audioless 
block in the case of a picture frame, stream, or audio 
designated by an editor or user. 
[0091] To add a priority in picture or audio frames for 



defining the prcx^essing priority under overioad. the fol- 
lowing methods are considered: a method for adding a 
priority to a communication header and a method for 
embedding a priority in the header of a bit stream in 

5 which a video or audio is encoded under encoding. The 
former makes it possible to obtain the infbrmaticxi for 
priority without decoding it and the latter makes it possi- 
ble to independently handle a single bit stream without 
depending on a system. 

10 [0092] When one picture frame (e,g. intra-frame 
encoded l-frame or inter-frame encoded P- or B-frame) 
is divided into a plurality of transmission packets, a pri- 
ority is added only to a communication header for trans- 
mitting the head of a picture frgime accessit>le as 

IS independent information in the case of a picture (when 
priorities are equal in the same picture frame, it is pos- 
sible to assume tfiat the priorities are not changed 
before the head of the next accessfole picture frame 
appears). 

20 [0093] Moreover, it is possfole to realize configuration 
in accordarK^e with control information by making the 
range of a value capable of expressing a priority varia- 
ble (for example, expressing time infbnmatfon with 16 
bits or 32 bits depending on the purpose). 

25 [0094] Furthermore, in the case of a decoder, priority 
deciding means for deciding a processing method is 
provkJed for a receiving terminal unit in accordance witii 
ttie priority under overioad of received various encoded 
pieces of information (see Rgure 47). 

30 [0095] Rgure 47 is an illustration for interpreting Prfor- 
Ities added to a picture and an audio and explaining pri- 
ority deciding means 5301 for deciding whether to 
perform decoding, 

[0096] That is. as shown In Figure 47, the priorities 
35 include a priority added to each stream of each picture 
or audio and a priority added to each frame of a picture 
or audio. It is possible to use these priorities independ- 
ently or by making a frame priority correspond to a 
stream priority The priority deckJing means 5301 
40 deckies a stream or frame to be decoded in accordance 
with these priorities. 

[0097] Deccxiing is performed by using two types of 
priorities for deciding a pr<x:essing priority under over- 
load at a terminal. 

45 [0098] That is. a stream priority (inter-time-series pri- 
ority) for defining a relative priority between bit streams 
such as a picture and audio and a frame priority Ontra- 
time-series priority) for defining a relative priority 
between decoding units such as picture frames In the 

so same stream are defined (Rgure 24). 

[0099] me former stream priority makes it possible to 
handle a plurality of vkleos or audios. The latter frame 
priority makes it possible to change scenes or add dif- 
ferent priorities even to the same intra-frame encoded 

55 picture frames (l-frame) in accordance with the intention 
of an editor. 

[0100] By making a stream priority correspond to a 
time assigned to an operating system (OS) for encoding 
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or decoding a picture or audio or a processing priority 
and thereby controlling the stream priority it is possible 
to control a processing time at an OS level. For exam- 
ple, in the case of Windowvs95/NT of Miaosoft Corpora- 
tion, a priority can be defined at five OS levels. By 5 
realizing encoding or decoding means by software in 
threads, it is possible to decide a priority at an OS level 
to be assigned to each thread in accordance with the 
stream priority of a purposed stream. 
[0101] The frame priority and stream priority 10 
described above can be applied to a transmission 
medium or data-recording medium. For exanple. by 
defining the priority of a packet to be transmitted as an 
access unit priority, it is possible to dedde a priority con- 
cemed with packet transmission or a priority for is 
processing by a terminal under overioad in accordance 
with the relation between frame priority and stream pri- 
ority such as the relation of Access Unit Priority = 
Stream Priority - Rame Priority. 

[0102] Moreover, it is possible to dedde a priority by 20 
using a floppy disk or optical disk as a data-recording 
medium. Furthermore, it is possible to decide a priority 
by using not only a recording medium but also an object 
capable of recording a program such as an IC card or 
r%)M cassette. Furthermore, it is possible to use a 25 
repeater for a picture or audio such as a router or gate- 
way for relaying data. 

[0103] As a specifk; method for using a priority, when 
a receiving terminal is overioaded, priority deciding 
means for deciding the threshokJ of the priority of 30 
encoded information to be processed is set to a picture- 
extension control section or audio-extension control 
section and the time to be displayed (PTS) is compared 
with the elapsed time after start of processing or the 
time to be decoded (DTS) is compared with the time 35 
elapsed time after start of processing to change thresh- 
olds of the priority of encoded information to be proc- 
essed in accordance with the comparison result (it is 
also possible to refer to the insertion interval of l-frame 
or the grading of a priority as the information for chang- 40 
ing tiireshokls). 

[0104] In the case of the example shown in Figure 
20(a). a picture with tiie size of captured QCIF or GIF is 
encoded by an encoder (H.263) under encoding to out- 
put a time stamp (PTS) showing tiie time for decoding 45 
(DTS) or the time for displaying the picture, priority infor- 
mation showing processing sequence under overload 
(CGD. Computational Graceful Degradation), frame 
type (SN). and sequence numt>er together witii 
encoded information. 50 
[0105] Moreover, in the case of tiie example shown in 
Figure 20(b), an audio is also recorded through a micro- 
phone and encoded by an encoder (G.721) to output a 
time stamp (PTS) showing tiie time for decoding (DTS) 
or the time for reproducing an audio, priority information ss 
(CGD). and sequence number (SISQ together with 
encoded informatfon. 

[0106] Under decoding, as shown in Figure 20(c). a 



picture and an audio are supplied to separate buffers to 
compare ttieir respective DTS (decoding time) with the 
elapsed time after start of processing. When DTS is not 
delayed, tiie picture and ttie audio are supplied to their 
con^esponding decoders (H.263 and G.721). 
[01 07] The example in Figure 21 describes a method 
for addirig a priority by an encoder under overioad. For 
a picture, high priorities of "0" and "1 " are assigned to I- 
frame (intra-frame encoded picture frame) (the smaller 
a numerical becomes, the lower a priority becomes). P- 
frame has a priority of "2" which Is lower than that of I- 
frame. Because two levels of priorities are assigned to I- 
firame. it is possible to reproduce only l-frame having a 
priority of "0" when a terminal for decoding has a large 
load. Moreover, it is necessary to acQust the insertion 
interval of l-frame in accordance with a priority adding 
method. 

[0108] The example in Figure 22 shows an illustration 
showing a method for deciding a priority at a receiving 
terminal under overload. The priority of a frame to be 
disused is set to a value larger tiign a cutOffPriority. 
That is, every picture frame is assumed as an object to 
be processed. It is possible to previously know the max- 
imum value of priorities added to picture frames by com- 
municating it from the transmitting side to the receiving 
side (step 101). 

[0109] When DTS is conipared with the elapsed time 
after start of processing and resultantly. the elapsed 
time is larger tiian DTS (when decoding is not in time), 
the threshold of the priority of a picture or audio to be 
processed is decreased to ttiin out processings (step 
102). However, when the elapsed time after start of 
processing is smaller than DTS (decoding is in time), 
the threshold of a priority is increased in order to 
increase the number of pictures or audio which can be 
processed (step 103). 

[01 1 0] If the image from one before is skipped by P- 
frame. no processing is performed. If not. a priority off- 
set value is added to the priority of a picture frame (or 
audio frame) to compare the priority offset value with 
tiie threshold of the priority. When the offset value does 
not exceed tiie threshold, data to be decoded is sup- 
plied to a decoder (step 1 04). 

[01 1 1 ] A priority offset allows the usage of previously 
checking the performance of a machine and communi- 
cating the offset to a receiving terminal (tt is also possi- 
ble that a user issues designation at the receiving 
terminal) and tiie usage of changing priorities of a plu- 
rality of video and audio streams in streams (for exam- 
ple, thinning out processings by increasing the offset 
value of the rearmost backgrourvJ). 
[0112] When a multi-stream is purposed, it is also 
possible to add a priority for each stream and decide the 
skip of decoding of a picture or audio. Moreover, in tiie 
case of real time communication, it is possft)le to decide 
whether decoding is advanced or delayed at the termi- 
nal by handling the TR (Temporary Reference) of H.283 
similarly to DTS and realize the skipping same as 
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described above. 

[0113] Rgure 23 is an illustration showing tenrqx)ral 
change of priorities by using the above algorithm. 
[0114] Figure 23 shows the change of a priority to be 
added to a picture frame. This priority is a priority for s 
deciding whether to perform decoding when a temiinal 
is overioaded, which is added every frame. The smaller 
the value of a priority becomes, the higher the priority 
becomes. In the case of the example in Rgure 23. 0 has 
the highest priority. When the threshold of a priority is 3. io 
a frame having a priority to which a value larger than 3 
is added is disused without being decoded and a frame 
having a priority to which a value of 3 or less is added is 
decoded. By selectively discussing frames in accord- 
ance with priorities, it is pc^t>le to control the load of a is 
terminal. It is also possible to dynamically decide the 
priority threshold In accordance with the relation 
between flie present processing time and the decoding 
time (DTS) to be added to each frame. This technique 
can be applied not only to a picture frame but also to an 20 
audio in accordance with the same procedure. 
101 1 5] In tiie case of a transmission line such as inter- 
net when it is necessary to retransmit encoded infor- 
mation lost under transnussion. it is possible to 
retransmit only a picture or audio required by the receiv- 25 
ing side by providing a retransmission request priority 
deciding section for deciding the threshold of tiie priority 
of the encoded information to be retransmitted for a 
reception control section and deciding the threshold of 
the priority added to the encoded information whose 30 
retransmission should be requested in accordance with 
the information for priority, retransmission frequervsy. 
loss rate of information, insertion interval of intra-frame 
encoded frame, grading of priority (e.g. five-level prior- 
ity) which are controlled by the priority deciding section. 35 
If the retransmission frequency or loss rate of informa- 
tion is too large, it is necessary to raise the priority of the 
information to be retransmitted and lower the retrans- 
mission or loss rate. Moreover, by knowing the priority 
used for ttie priority deciding section, it is possible to 4o 
prevent the information to be processed from being 
transmitted. 

[01 1 6] In the case of a transmitting terminal, when an 
actual transfer rate exceeds the target transfer rate of 
the information of the transmitting terminal or when writ- 4S 
ing of the encoded information into a transmitting buffer 
is delayed as the result of comparing tiie elapsed time 
after start of transfer processing with the time added to 
the encoded information to be decoded or displayed, it 
is possible to transmit a picture or audio matching with so 
the target rate by using a priority added to encoded 
information and used by the priority deciding section of 
the receiving terminal when the terminal is overioaded 
and thereby thinning out transmissions of information. 
Moreover, by introducing the processing skipping tunc- ss 
tion under overload performed at the receiving-side ter- 
minal into the transmrtting-slde terminal, it is possQ3le to 
control a failure due to overioad of the transmitting-side 



terminal. 

[0117] By making it possble to transmit only neces- 
sary infbmriation out of the above-described AL informa- 
tion according to necessity, it is possitrie to adjust the 
amount of information to be transmitted to a narrow- 
band communication channel such as an analog tele- 
phone line. It is possible to reconrtblne the AL informa- 
tion (data control information) used for the trar^mrtting 
side by deciding the data control information to be 
added to data at a transmitting-side terrrunal before 
transmitting the data, communicating the data control 
Information to be used to a receiving terminal as control 
information (for exarrple, using only a random access 
flag), and rewriting at the receiving-side terminal teased 
on the obtained control information tiie information 
about a transmission structure (showing which AL infor- 
mation is used) stored in the transmission format storing 
section 103 (see Rgure 16). 

[0118] Rgure 4 is an illustration for explaining a 
method for dynamically changing header information 
added to the data for a picture or audio to be transmit- 
ted. In the case of the example in Figure 4. the data 
(ES) to be transmitted is deconposed into data pieces 
and the identifying information (sequence number) for 
showing tiie sequence of data, the information (marker 
bit) showing whether it is a start position capable of 
processing data pieces, and time information (time 
stanp) concerned with transfer of data pieces are 
added to data pieces in the form of communication 
headers by assuming that the above pieces of informa- 
tion con-espond to transmission control information of 
the present invention. 

[0119] Specifically. RTP (Realtime Transfer Protocol. 
REC1889) uses the information for the above sequence 
number, marker bit. time stanp, object ID (referred to as 
SSRC). and version nun^er as communication head- 
ers. Though a header-information item can be 
extended, the above items are always added as fixed 
items. However, when the realtime communication such 
as the case of a video telephone and transmission of 
accumulated media such as the case of video-on- 
demand are present togetiier in an environment in 
which a plurality of different encoded pictures or audio 
are simultaneously transmitted, identifying means is 
necessary because meanings of communication head- 
ers are different from each other. 
[0120] R>r example, time-stamp information shows 
PTS that is a reproducing time as previously described 
in the case of MPEGl/2. In the case of H.261 or H.263, 
however, the time-stamp information shows a time inter- 
val when tiie information is encoded. However, to proc- 
ess H.263 synchronously witti an audio, it is necessary 
to show that a time stamp is PTS information. This is 
because time-stamp information shows the time interval 
between encoded frames in tiie case of H. 263 and it is 
d^ined by RTP tiiat tiie time stamp of the first frame is 
random. 

[0121] Therefore, it is necessary to add a flag showing 
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whether a time stamp is PTS as (a) communication 
header information (it is necessary to extend a commu- 
nication header) or (b) header information for payload of 
H.263 or H.261 (that is. AL information) (in this case, it 
is necessary to extend payload information). 5 
[0122] A marl^r bit servirig as the information show- 
ing whether it is a start position capable of processing 
data pieces is added as RTP header information. More- 
over, as described above, there is a case in which it is 
necessary to provide an access flag showing that it is a 10 
start jx)sition capable of accessing data and a random 
access flag showing that it is possible to access data at 
random for AL information. Because doubly providing 
flags for a communication header lowers the efficiency, 
a method of substituting an AL flag by a flag prepared is 
for the commioiication header is also considered. 

(c) The problem is solved by newly providing a flag 
showing that an AL flag is substituted by the header 
added to a communication header without adding a 20 
flag to AL for the communication header or defining 
that the marker bit of the communication header is 
the same as that of AL (it is expected that interpre- 
tation can be quicMy performed cbrrpared to the 
case of providing a flag for AL). That is. a flag is 2s 
used which shows whether the marker bit has the 
same meaning as the flag of AL. In this case, it is 
considered to improve the communication header 
or describe it in an extension regioa 

30 

[0123] However, (d) it is also possible to interpret the 
meaning of the marker bit of the communication header 
so as to mean that at least either of a random access 
flag and an access flag is present in AL In tiiis case, it 
is possible to know that the meaning of interpretation is 35 
changed from the conventional case by tiie versfon 
number of the communication header. Moreover, 
processing is simplified by providing an access flag or 
random access flag only for the comnrunication header 
or the header of AL (for the former, a case of providing 40 
the flag for both the headers is considered txjt it is nec- 
essary to newly extend the communicatfon header). 
[0124] It is already described to add the information 
showing the priority of data processing as the informa- 
tion for AL. By adding the data-processing priority to the 4S 
communication header, it is possible to decide the 
processing of the data-processing priority without inter- 
preting the contents of data also on a network. Moreo- 
ver, in the case of IPv6, it is possible to add the priority 
at a layer lower ttian the level of RTP. so 
[0125] By adding a timer or counter for showing the 
effective period of data processing to the communica- 
tion header of RTP. it is possible to decide how the state 
of a transmitted packet changes. For example, when 
necessary decoder software is stored in a memory hav- ss 
tng a low access speed, it is possible to decide the infor- 
mation required by a decoder and when the information 
is required by a timer or counter. In this case, the infor- 



mation for the priority of a timer or counter or tiie infor- 
mation for the priority of data processing is unnecessary 
for AL information depending on the purpose. 
[0126] Figures 5(a) and 5(b) and Figures 6(a) to 6(d) 
are illustrations for explaining a method for adding AL 
information. 

[01 27] By sending the control infbrmatfon for commu- 
ntoating whether to add AL to only the head of the data 
to be transmitted as shown in Figure 5(a) or whether to 
add AL to each data piece after deconposing the data 
to be transmitted (ES) into one data piece or more to a 
receiving terminal as shown in Figure 5(b). it is possible 
to select the grading for handling transmission informa- 
tion. Adding AL to subdivided data is effective when 
access delay is a problem. 

[0128] As described above, to previously communi- 
cate recombination of data control information at the 
receiving side or change of methods for arrariging data 
control information to data to a receiving-side terminal, 
receiving-terminal con-espondence can be smootiily 
performed by using the expression of a flag, counter, or 
timer and thereby, preparing the expression as AL infor- 
mation or as a communfoation header to communicate 
it to the receiving terminal. 

[0129] In the case of tiie above examples, a method 
for avoiding duplication of the header of ffTTP (or com- 
munication header) witii AL information and a method 
for extending the communication header of RTP or AL 
information are described. However, it is not always 
necessary for tiie present invention to use RTP. For 
example, it is possit)le to newly define an original com- 
munication header or AL information by using UDP or 
TCP. Though the internet profile uses RTP sometimes, 
a multifunctibna) header such as RTP is not defined in 
tiie Raw profile. The following four types of concepts are 
considered for AL information and communication 
header (see Rgures 6(a) to 6(d)). 

(1) The header information of RTP or AL informa- 
tion is corrected and extended so that the header 
information already assigned to RTP and that 
already assigned to AL are not overiapped (particu- 
lariy. the information for a time stamp is overlapped 
and the priority information for a timer, counter, or 
data processing k>ecorhes extension information). 
Or. it is possible to use a method of not extending 
the header of RTP or not considering duplication of 
AL information with information of RTP, They corre- 
spond to the contents having been shown so far. 
Because a part of RTP is already practically used 
for H.323. it is effective to extend RTP having com- 
patibility. (See Figure 6(a).) 

(2) Independently of RTP, a communication header 
is simplified (for example, using orily a sequence 
number) and remainder is provkled for AL informa- 
tion as multifunctional control information. Moreo- 
ver, by making it possible to variably set items used 
for AL information before communication, it is pos- 
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sible to specify a f lexibie transmission format. (See 
Rgure 6(b).) 

(3) Independently of RTP, AL information is simpli- 
fied (for an extreme example, no information is 
added to AL) and every control inf(»rmation is pro- 
vided for a communication header. A sequence 
number, time st amp , mariner bit payload type, and 
object ID frequently used as communication head* 
ers are kept as fixed information and data-process- 
ing priority information and timer information are 
respectively provided with an identifier showing 
whether extended information is present as 
extended information to refer to the extended infor- 
mation if the information is defined. (See Figure 
6(c).) 

(4) Independentiy of RTP, a communication header 
and AL information are sinplif ied and a format is 
defined as a packet separate from the communica- 
tion header or AL information to transmit the format 
For example, a method is also considered in which 
only a marker bit. time stamp, and object ID are 
defined for AL information, only a sequence 
number is defined for a communication header, and 
payload information, data-processing priority infor- 
mation, and timer information are defined as a 
transmission packet (second packet) separate from 
the above information and transmitted. (See Rgure 
6(d).) 

[0130] As described above, when considering a pur- 
pose and header information already added to a picture 
or audio, it is preferable so as to be at>le to freely define 
(customize) a packet (second packet) to be transmitted 
separately from a conrununication header. AL informa- 
tion, or data in accordance witii the purpose 
[0131] Rgure 7 is an illustration for explaining a 
method for transmitting information by dynamically mul- 
tiplexing and separating a plurality of logical transmis- 
sion lines. The number of logical transmission lines can 
be decreased by providing an information multiplexing 
section capable of starting or ending multiplexing of the 
information for logical transmission lines for transmitting 
a plurality of pieces of data or control information in 
accordance with the designation by a user or the 
number of logical transmission lines for a transmitting 
section and an information separating section for sepa- 
rating multiplexed information for a reception control 
section. 

[01 32] In Rgure 7, the information multiplexing section 
is referred to as Xaroup MUX" and specifically, it is pos- 
sible to use a multiplexing system such as H.223. !t Is 
possible to provide the GroMp MUX for a transmit- 
ting/ireceiving terminal. By providing the Group MUX for 
a relay router or terminal, it is possible to correspond to 
a narrow-band communication chsmnel. Moreover, by 
realizing Group MUX with H.223, it is possble to inter- 
connect H.223 and H.324. 

[0133] To quickly fetch the control information (multi- 
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plexing control information) for the information multi- 
plexing section, it is possible to reduce a delay due to 
multiplexing by transmitting the control information in 
the information multiplexing section through another 

5 logical ti'ansmission line without multiplexing the control 
information with data by the infomiation multiplexing 
section. Thereby, it is possfole for a user to select 
whether to keep the consistency with conventional mul- 
tiplexing or reduce a delay due to multiplexing by oom- 

10 municating and transmitting whether to multiplex the 
control information concerned with the information mul- 
tiplexing section with data and transmit them or transmit 
the control information through another logical transmis- 
sion line witiiout multiplexing the information with the 

IS data. In tiiis case, the multiplexing control information 
concemed witii the information multiplexing section Is 
Infonmation showing the content of multiplexing about 
how the information multiplexing section performs multi- 
plexing for each piece of data. 

20 [01 34] As described above, similarly, it is possible to 
transmit the notification of a method for transmitting at 
least the information for communicating the start and 
end of multiplexing, information for communicating the 
comkxnation of logical transmission lines to be multi- 

25 plexed, and control information concemed with multi- 
plexing (multiplexing control information) as control 
information in accordance with an expression method 
such as a flag, counter, or timer or reduce the setup time 
at the receiving side by transmitting data control infor- 

30 mation to a receiving-side terminal togetiier with data. 
Moreover, as previously described, it is possik)le to pro- 
vide an item for expressing a flag, counter, or timer for 
the transmission header of RTP. 
[0135] When a plurality of information nultiplexing 

35 sections or a plurality of information separating sections 
are present, it is possible to identify to which information 
multiplexing section the control information (multiplex- 
ing control information) belongs by transmitting the con- 
trol information (multiplexing control information) 

40 together with an identifier for identifying an inforrnation 
multiplexing section or information separating section. 
The control information (multiplexing control informa- 
tion) includes a multiplexing pattern. Moreover, by using 
a table of random nuodDer and thereby, deciding an 

45 identifier of an information multiplexing section or infor- 
mation separating section between terminals, it is pos- 
sible to generate an identifier of the information 
multiplexing section. For Sample, it is possible to gen- 
erate random numbers in a range determined between 

so ti^nsmitting and receiving terminals arxi use the largest 
veJiue for the identifier (identification nurnber) of the 
information multiplexing section. 
[01 36] Because the data multiplexed by the informa- 
tion multiplexing section is conventionally different from 

55 the media type defined in RTP. it is necessary to define 
the information showing that it is information multiplexed 
by the information multiplexing section (new media type 
H.223 is defined) for the payload type of RTP. 
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[01 37] By arranging the information to be transmitted 
by or recorded in the information multiplexing section In 
the sequence of control Information and data informa- 
tion so as to improve the access speed to multiplexed 
data, it is expected to quickly analyze multiplexed infor- 
mation. Moreover, it is possible to quickly analyze 
header information by fixing an item which is described 
. in accordance with the data corrtroi information added to 
control information and adding arxl multiplexing an 
identifier (unique pattern) different from data. 
[0138] Figure 8 is an illustration for explaining the 
transmission procedure of a broadcasting program. By 
using the relation between the kientifier of a logical 
transmission line and the dentifier of a broadcasting 
program as the information of the broadcasting program 
and thereby, transmitting control infoimation or adding 
the Identifier of a broadcasting program to data as data 
control information (AL information), it is possible to 
identify that the data transmitted through a plurality of 
transmission lines is broadcasted for which program. 
Moreover, by transmitting the relation between the iden- 
tifier of data (SSRC in the case of RTP) and the identi- 
fier of a logical transnnissfon line (e.g. port number of 
LAN) to a receiving-side terminal as control information 
and transmitting corresponding data after it is confirmed 
that the control information can be received by the 
receiving-skJe terminal (Ack/Reject). it is possit)le to 
form the correspondence between data pieces even if 
control information and data are respectively transmit- 
ted through an independent transmission line. 
[0139] By combining an kientifier showing the trans- 
mission sequence of broadcasting programs or data 
pieces with the information for a counter or timer for 
showing a term of validity in which broadcasting pro- 
gram or data can be used as information, adding the 
conrA>ined kientifier and information to the broadcasting 
program or data, and transmitting them, it is possible to 
realize broadcasting without return channel (when the 
term of validity almost expires, reproduction of the infor- 
mation or data for a broadcasting program is started 
even if information is insufficient). Moreover, a method 
can be considered in which control information and data 
are ftxoadcasted without toeing separated from each 
other by using the address of a single communication 
port (multicast address). 

[0140] . In the case of communication with no t>ack 
channel, it is necessary to transmit control information 
sufFfoiently before transmitting data so as to enat)le the 
receiving terminal to know a structural information of 
data. Moreover, control information shouki k>e transmit- 
ted through a transmission channel free from packet 
loss and having a high reliability. However, when using a 
transmission channel having a low reliability, it Is neces- 
sary to cyclically transmit the control information having 
the same transmission sequence number. This is not 
restricted to the case of transmitting the control informa- 
tion concerned with a setup time. 
[0141] Moreover, it is possible to flexibly control and 



transmit data by selecting an item which can be added 
as data control information (e.g. access flag, random 
access flag, data reproducing time (PTS). or data- 
processing-priority information), deciding whether to 

5 transmit the data control information together with the 
kientifier (SSRC) of data as control information through 
a logical transmission line different from that of the data 
or transmit the data control information as data control 
information (information for AL) together witti the data at 

10 the transmitting skle before transmitting the data, and 
communicating and transmitting the data to the receiv- 
ing side as control information. 
101 42] Thereby, it is possible to transmit data informa- 
tion without adding information to AL. Therefore, to 

15 transmit the data for a picture or audio by using RTP. it 
is unnecessary to extend the definition of the pc^load 
having been defined so far. 

[0143] Figures 9(a) and 9(b) are illustrations showing 
a picture or audio transmission method considering the 

20 read time and rise time of prograrn or data. Particularly, 
when the resources of a terminal are limited like the 
case of satellite txoadcasting or a portable terminal 
having no return channel and being unkllrectional. pro- 
gram or data is present and used at a receiving-skie ter- 

25 minal, a necessary program (e.g. H^63. MPEG1/2, or 
software of audio decoder) or data (e.g. vkJeo data or 
audio data) is preserrt in a memory (e.g. DVD, hard disk, 
or file server on networl^ requiring a lot of read time, it 
is possible to reduce the setup time of program or data 

30 required in advance by previously receiving it as control 
information or receiving it together with data as data 
control information in accordance with the expression 
metiiod such as the identifier for Identifying the program 
or data, kientifier (e.g. SSRC. or Logical Channel 

35 Number) of a stream to be transmitted, or a flag, counter 
(count-up/down), or timer for estimating tiie point of time 
necessary for a receiving terminal (Figure 18). 
[01 44] When program or data is transmitted, by trans- 
mitting the program or data from the trar^mitting side 

40 together vnth the information showing the storage desti- 
nation (e.g. hard disk or memory) of the program or data 
at a receiving terminal, time required for start or read, 
relation t)6tween tiie type or storage destination of a ter- 
minal and the time required for start or read (e.g. rela- 

45 tion tsetween CPU power, storage devk^e, and average 
response time), and utilization sequence, it is possit)le 
to schedule the storage destination and read time of the 
program or data if the program or data necessary for the 
receiving terminal is actually required. 

so [0145] Figures 10(a) and 10(b) are illustrations for 
explaining a method for corresponding to zapping 
(channel change of TV). 

[0146] When it is necessary to execute a program at 
a recaving terminal differentiy from the case of conven- 
ss tional satellite broadcasting for receiving only pictures, 
tiie setup time until the program is read and started is a 
large problem. The same is true for the case in which 
available resources are limited like tiie case of a porta- 
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ble terminal. 

[0147] It Is expected that the setup time at a receiving- 
side terminal can be decreased by (a) using a main 
looking-listening section by which the user looks at and 
listens to, and an auxiliary looking-listening section in 
which a receiving terminal cyclically monitors programs 
other th€ui the program looked and listened by a user 
and receiving the relation t>etween kJentif ier for identify- 
ing program or data required in advance, information for 
a flag, counter, or timer for estimating the point of time 
necessary for the receiving terminal, and program as 
control information (information transmitted by a packet 
different from that of data to control terminal processing) 
or as data control information (information for AL), and 
preparing read of the program or data together with data 
as one of the settlement measures when to program or 
data necessary for a program other than the program 
looked and listened by the user is present in a memory 
requiring a lot of time for read. 
[0148] It is possit)le to prevent a screen from stopping 
under setup by setting a broadcasting channel for 
broadcasting only heading pictures of the pictures 
broadcasted through a plurality of channels and switch- 
ing programs by a user, and thereby, when necessary 
program or data is present in a memory requiring a lot 
of time for read, temporarily selecting the fieading pic- 
ture of a program required by the user and showing it for 
the user or showing that program or data is currently 
read, and restarting the program required by the user 
after necessary program or data Is read by the memory 
as the second one of the settlement measures. The 
atxDve heading pictures include broadcasted pictures 
obtained by cyclically sanpiing programs broadcasted 
through a plurality of cheinnels. 
[0149] Moreover, a timer is a time expression and 
shows the point of time when a program necessary to 
decode a data stream sent from the transmitting side is 
necessary. A counter is the t>asic time unit determined 
between transmitting and receiving terminals, which 
can be information showing what-th time. A flag is trans- 
mitted and communicated togetiier with the data trans- 
mitted before the time necessary for setup or control 
information (information transmitted through a packet 
different from that of data to control terminal process- 
ing). It is possfole to transmit the timer and counter by 
enU^edding them in data or transmit them as control 
information. 

[0150] Furthermore, to decide a setip time, the time 
in which setup is performed can be estimated by. when 
using a transnv'ssion line such as ISDN operating on the 
clock base* using a transmission serial number for iden- 
tifying a transmission sequence as transmission control 
information in order to communicate from the transmit- 
ting terminal to the receiving terriiinal a time point when 
program or data is required and thereby communicating 
the serial nunrt>er to a receiving terminal together with 
data as data control information or as control informa- 
tion. Furthermore, when a transmission time is f luctu- 
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ated due to jitter or delay like internet, it is necessary to 
add the trarismission time to the setup time by conskJer- 
ing the propagation delay of transmission in accordance 
with jitter or delay time by the means for realizing RTCP 
5 (media transmission protocol used for internet). 

[01 51 ] Figures 1 1 (a) to 1 9(b) are illustrations showing 
specific examples of protocols actually transferred 
between terminals. 

[0152] A transmission format and a transmission pro- 
10 cedure are described in ASN. 1 . Moreover, the transmis- 
sion format is exterxled on the basis of H.245 of ITU. As 
shown in Figure 11(a), objects of a picture and audio 
can have a hierarchical structure. In the case of this 
example, each object ID has flie attributes of a broad- 
15 casting-program identifier (program ID) and an object ID 
(S SRC) and the.structural information and synthesizing 
method between pictures are descrH^ed by a script lan- 
guage such as Java or VRML 

[0153] Figure 1 1(a) is an illustration showing exarn- 

20 pies of the relation between objects. 

[0154] In Figure 1 1(a), objects are media such as an 
audio-vkJeo, CG. and text In the case of the examples 
in Figure 11(a), objects constitute a hierarchical struc- 
ture. Each object has a program number "Program ID" 

25 correspoTKltng to TV channel) and an object identifier 
"Object ID" for identifying an object When transmitGng 
each object in accordance with RTF (media transmis- 
sion protocol for transmitting media used for internet. 
Realtime Transfer Protocol), it is posstt^le to easily iden- 

30 tify the object by making the object identifier correspond 
to SSRC (synchronous source identifier). Moreover, it is 
possilsle to descril>e the structure between ol^ects with 
a description language such as JAVA or VRML 
[0155] Two types of methods for transmitting the 

35 objects are considered. One is the broadcasting type in 
which the objects are unilaterally transmitted from a 
transmitting-side terminal. The other is the type (com- 
munication type) for transferring the objects between 
transmitting arvl receiving terminals (terminals A and 

40 B), 

[0156] For exanrple. it is possible to use RTP as a 
transmission method in the case of internet Control 
information is transmitted by using a transmission chan- 
nel referred to as LCNO in the case of the starxiard for 
45 video telephones. In the case of the example in Figure 
1 1(a). a plurality of transmission channels are used for 
transmission. The same program channel (program ID) 
is assigned to these channels. 

[01 57] Figure 1 1 (b) is an illustration for explaining how 
50 to realize a protocol for reafizing the functions described 
for the present inverrHon. The transmission protoGo! 
(H.245) used for the video-telephone standards (H.324 
and H.323) is described below. The functfons described 
for the present invention are realized by extending 
55 H.245. 

[01 58] The description method shown by thlB example 
in Rgure 11(b) is the protocol descr'qation method 
referred to as ASN.1. "Terminal Capability Set" 
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expresses the performance of a terminal. In the case of 
the example in Figure 11(b). the function described as 
**mpeg4 Capability" is extended for the conventional 
H.245. 

(01 59] In Figure 1 2. "mpeg4 CapatMllty" describes the s 
maximum number of pictures "Max Nuntjer Of pictures" 
and the maximum number of audio ("Max Nunr^er Of 
Audio") which can be simultaneously processed by a 
terminal and the maximum number of multiplexing tunc- 
tions ("Max Number Of Mux") which can be realized by io 
a terminal. 

C01 60] In Figure 1 2. these are expressed as the max- 
imum number of objects ("Number Of Process Object") 
which can be processed. Moreover, a flag showing 
whether a comnrainication header (expressed as AL in is 
Rgure 12) can be changed is described. When the 
value of the flag is true, the conrmunication header can 
be changed. To communicate the numtier of objects 
which can be processed between terminals to each 
other by using "MPEG4 Capability", the communicated 20 
side returns "MEPG4 Capability Ack" to a terminal from 
which "MEP04 Capability" is transmitted if the commu- 
nicated side can accept (process) the objects txjt 
returns "MEPG4 Capability Reject" to the terminal if not. 
[01 61 ] Figure 1 3(a) shows how to describe a protocol 25 
for using the atx)ve Group MUX for multiplexing a plural- 
ity of logical channels to one transmission channel 
(transmissron channel of LAN in the case of this exam- 
ple) in order to share the transmission channel by logi- 
cal channels. In the case of the example in Figure 1 3(a), 30 
multiplexing means (Group MUX) is made to corre- 
spond to the transmission channel fLAN Port Number") 
of LAN (Ljocal Area Network). "Group Mux ID" is an 
identifier for identifying the> multiplexing means. To 
share the multiplexing means by terminals by using 35 
"Create Group Mux" and perform communication 
between the terminals, the communicated side returns 
"Create Group Mux Ack" to a terminal from which "Cre- 
ate Group Mux" is transmitted if the side can accept 
(use) the multiplexing means txit returns "Create Group 40 
Mux Reject" to the terminal if not. Separating means 
serving as means for performing an operation reverse to 
that of the multiplexing means can be realized by the 
same method. 

[0162] In Figure 1 3(b), a case of deleting already-gen- 45 
erated multiplexing means Is descrit)ed. 
[0163] In Figure 1 3(c), the relation between the trans- 
mission channel of LAN and a plurality of logical chan- 
nels is descril^. 

[0164] The transmission channel of LAN is described so 
in accordance witii "LAN Port Number" and the logical 
channels are described in accordance with "Logical 
Port Number. 

[0165] In the case of the examples in Figure 13(c), it 
is possible to make the transmission channel of one ss 
LAN correspond to up to 1 5 logical channels. 
[0166] In Figure 13, when the number of MUXs that 
can be used is only one, Group Mux ID is unnecessary 



Moreover, to use a plurality of Muxes, Group Mux ID is 
necessary tor each command of H.223. Furthermore, it 
is possible to use a flag tor communicating the relation 
between ports used between the multiplexing means 
and separating means. Furttiermore, it is possitDle to 
use a command making it possible to select whether to 
multiplex control information or transmit the information 
tiirough another fogical transmission line. 
[0167] In the case of tiie explanation in Figures 13(a) 
to 13(c), the transmission channel uses LAN. However, 
it is also possible to use a system using no internet pro- 
tocol like H.223 or fAPEG2, 

[0168] In Figure 14, "Open Logical Channel" shows 
the protocol description for defining the attribute of a 
transmission channel. In the case of the example in Fig- 
ure 14. "MPEG4 Logical Channel Parameters" is 
extended and defined for the protocol of H.245. 
[01 69] Rgure 1 5 shows that a program number (cor- 
responding to a TV channel) and a program name are 
made to con^espond to the transmission channel of LAN 
("MPEG4 Logical Channel Parameters"). 
[0170] Moreover, in Figure 15, "Broadcast Channel 
Program" denotes a description method for transmitting 
tfie correspondence between LAN transmission chan- 
nel and program nun^er In accordance with the broad- 
casting type. The exanple in Figure 15 makes it 
possible to transmit tiie conrespondence between up to 
1.023 transmission channels and program numt>ers. 
Because transmission is unilaterally performed from the 
transmitting side to the receiving side in the case of 
broadcasting, it is necessary to cyclically transmit these 
pieces of information by considering tiie loss during 
transmission. 

[01 71 ] In Figure 1 6(a), the ath-ibute of an object (e,g. 
pk:ture or audio) to be transmitted as a program is 
descrtoed ("MPEG4 Object Classdefinition"). Object 
information ("Object Structure Element") is made to cor- 
respond to a program identifier ("Program ID"). It is pos- 
sible to make up to 1,023 objects correspond to 
program identifiers. As the object information, a LAN 
transmission channel ("LAN Port Number"), a flag 
showing whether scramble is used ("Scramble Flag"), a 
field for defining an offset value for changing the 
processing priority when a terminal is overloaded 
("CGD Offset), and an identifier (Media Type) for identi- 
fying a type of the media (picture or audio) to be trans- 
mitted are described. 

[01 72] In the case of the example in Figure 16(b), AL 
(in this case, defined as additional information neces- 
sary to decode pictures for one frame) is added to con- 
trol decoding of €S Cm this case, defined as a data string 
corresponding to pictures for one frame). As AL infor- 
mation, the following are defined. 

(1) Random Access Rag (flag showing whether to 
be independently reproducible, true for an intra- 
frame encoded picture frame) 

(2) Presentation Time stamp (time displayed by 
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frame) 

(3) CGD Priority (Value of priority for deciding 
processing priority when terminal is cverioaded) 

[0173] The example shows a case of transmitting the s 
data string for one frame by using RTP (protocol for 
transmitting continuous media through intemet Real- 
time Transfer Protocol). "AL Reconfiguration" is a trans- 
mission expression for changing the maximum value 
that can be expressed by the above AL io 
[01 74] The example in Figure 1 6(b) makes it possible 
to express up to 2 bits as "Random Access Flag Max 
Bit". For exanpie, when there is no bit. Random Access 
Flag is not used. When there are two bits, the maximum 
value is equal to 3. is 
[0175] Moreover, the expression with a real number 
part and a mantissa part is allowed (e.g. 3 6). When no 
data is set, an operation under the state decided by 
default is allowed. 

[0176] In Figure 17. "Setup Request" shows a trans- 20 
mission expression for transmitting a setup time. "Setup 
Request** is transmitted before a program is transmitted, 
a transmission channel nuni>er ("Logical Channel 
Numt>er") to be transmitted, a program ID ("execute 
Program Number) to be executed, a data ID ("data 25 
Number") to be used, and the ID of a command ("exe- 
cute Convnand Nunrt>er") to be executed are made to 
oon^espond to each other and transmitted to a receiving 
terminal. Moreover, an execution authorizing flag 
("flag"), a counter ("counter") describing whether to start 30 
execution when receiving Setup Request how many 
times, and a timer value ninier**) showing whether to 
start execution after how many hours pass can be used 
as other expression methods by making them corre- 
spond to transmission channel numbers. 35 
[0177] Rewriting of AL information and securing of 
rise time of Group Mux are listed as examples of 
requests to be demanded. 

[0178] Rgure 18 is an illustration for explaining a 
transmission expression for communicating whether to 40 
use the AL described for Figure 16(b) from a transmit- 
ting terminal to a receiving terminal CControl AL defini- 
tion"). 

[0179] In Rgure 18, if "Random Access Rag Use" is 
true. Random Access Flag is used. If not. it is not used. 45 
a is possitile to transmit the AL change notification as 
control information through a transmissfon channel sep- 
arate from that of data or transmit it through the trans- 
mission channel same as that of data together with the 
data. so 
[0180] A decoder program is listed as a program to be 
executed. Moreover, a setup request can be used for 
broadcasting and communication. Furthermore, which 
item serving as control information is used as Al infor- 
mation is designated to a receiving terminal in accord- ss 
ance with the above request Furthermore, it is possit>le 
to designate which item is used as communication 
header, which item is used as AL information and which 



item is used as control informatfon to a receiving termi- 
nal. 

[0181] Rgure 19(a) shows the example of a transmis- 
sion expression for changing the structure of header 
information (data control information, transmission con- 
trol information, and control information) to be transmit- 
ted by using an information frame identifier ("header 
ID") between transmitting and receiving terrrdnats in 
accordance with the purpose. 

[0182] In Rgure 19(a), "class ES header" separates 
the structure of the data control information to t>e trcms- 
mitted through a transnr^ssion channel same as that of 
data from ti^at of the information with which transmis- 
sion control information is transmitted between tians- 
mitting and receiving terrrtinals in accordance with an 
.information frame identifier. 

[0183] For exanf^le. only the item of "buffer Size ES" 
is used when the value of "header ID" Is 0 but the item 
of "reserved" is added when the value of "header ID" is 
1, 

[0184] Moreover, by using a default identifier ("use 
Header Extension"), it is decided whether to use a 
default-type information frame. When "use Header 
Extension" is true, an item in an if-statement is used. It 
is assumed that these pieces of structural information 
are previously decided between transmitting and receiv- 
ing terminals. Furthermore, It is possible to use a struc- 
ture for using either of an information frame Identifier 
and a default identifier. 

[0185] In Rgure 19(b). "AL configuration" shows an 
example for changing the structure of control informa- 
tion to be transmitted tiirough a transmission channel 
different from tiiat of data between ti-ansmitting and 
receiving terminals in accordance with the purpose. The 
usage of an information frame Identifier and that of a 
default identifier are the same as the case of Rgure 
19(a). 

[0186] In tiie case of the present invention, methods 
for realizing a system for simultaneously synthesizing 
and displaying a plurality of pictures and a plurality of 
audio are specifically described from the folfowing view- 
points. 

(1) A method for transmitting (communicating and 
broadcasting) a picture and an audio through a plu- 
rality of logical tiansmission lines and controlling 
them. Particularly, a method for respectively trans- 
mitting control inforhnation and data through an 
Independent logical transmission line is described. 

(2) A method for dynamically changing header 
information (AL information) added to the data for a 
picture or audio to bei transmitted. 

(3) A metiiod for dynamically changing communica- 
tion header information added for transmission. 

Specifically, for Items (2) and (3), a method for 
uniting and controlling the information overiapped 
in AL information and communication header arxl a 
method for transmitting AL information as control 
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information are described. 
(4) A method for dynamically multiplexing and sep- 
aerating a plurality of logical transmission lines and 
transmitting inlbrmation. 

A method for economizing the number of chan- 
nels of transmission lines and a method fbr reeUiz- 
ing efficient multiplexing are described. 
{5i A method fbr reading a program or data and 
transmitting pictures and audio considering a rise 
time. Moreover, a method for reducing an apparent 
setup time for various functions and purposes is 
described. 

(6) A method fbr transmitting a picture or aucGo fbr 
zapping. 

[0187] The present invention is not restricted to only 
synthesis of two<limensional pictures. It is also possible 
to use an expression method of combining a two-dimen- 
sional picture with a three-dimensional picture or 
include a picture synthesizing method for synthesizing a 
plurality of pictures so that they are adjacent to each 
otfier like a wide-visual-field picture (panoramic picture). 
[0188] Moreover, the present invention does not pur- 
pose only such communication systems as bidirectional 
CATV and B-ISDN. Fbr example, it is possible to use 
radio waves (e.g. VHF band or UHF band) or a broad- 
casting satellite for transmission of pictures and audio 
from a center-side terminal to a home-side terminal and 
an analog telephone line or N-ISDN for transmission of 
information from a home-side terminal to a center-side 
terminal (it is not alvyays necessary that pictures, audio, 
and data are multiplexed). 

[0189] Moreover, it is possit)le to use a communication 
system using radio such as IrDA, PHS (Personal Handy 
f^hone). or radio LAN. Furthermore, a purposed termi- 
nal can be a portable terminal such as a portable infor- 
mation terminal or a desktop terminal such as a setup 
BOX or personal computer. Furthermore, a video tele- 
phone. muHipoirtt monitoring system, multimedia data- 
base retrieval system, and game are listed as 
application fields. The present invention includes not 
only a receiving terminal but also a server and a 
repeater to be connected to a receiving terminal. 
[0190] Furthermore, in the case of the above exam- 
ples, a method for avoiding the overlap of the (commu- 
nication) header of RTF with AL information and a 
method for extending the communication header of RTF 
or AL information are described. However, it is not 
always necessary for the present invention to use RTF. 
For exanple, it is also possible to newly define an origi- 
nal communication header or AL information by using 
UDF or TCP. Though an internet profile uses RTF 
sometimes, a multifunctional header such as RTF is not 
defined for a Raw profile. There are four types of con- 
cepts about AL information and communication header 
as described above. 

(01 91 ] Thus, by dynamically dedding the information 
frame of data control information, transmission control 



information, or control information used by the transmit- 
ting and receiving terminals (e.g. information frame 
. including the sequence of information to be added and 
the number of bits for firstly assigning a random access 
5 flag as 1 -bit flag information and secondly assigning 16 
bits in the form of a sequence numt)er). it is possible to 
change only an information frame coiresponding to ttie 
situation in accordance with the purpose or transmis- 
sion line. 

10 [01 92] The frame of each piece of information can be 
any one of the frames already shown in Figures 6(a) to 
6(d) and in the case of RTF, the data control information 
(AL) can be the header information for each medium 
(e.g. in the case of H.263, the header information of tiie 

IS video or tiiat of the payload intrinsic to H.263), transmis- 
sion control information can be the header information 
of RTF, and control information can be the information 
for controlling RTF such as RTCR 
[0193] Moreover, in tiie case of a publidy-known infor- 

20 mation frame previously set between transmitting and 
receiving terminals, by providing a default identifier for 
showing whether to process information by transmitting 
and receiving for data control information, transmission 
control information, and control information (information 

25 transmitted through a packet different from that of data 
to control terminal processing) respectively; it is possi- 
ble to know whether information frames are changed. 
By setting the default identifier and communicating the 
changed content (such as change of time stamp Infor- 

30 mation from 32 to 16 bits) only when change is per- 
formed in accordance witti the method shown in Figure 
16, it is prevented to transmit unnecessary configuration 
information even when frame information of information 
is not changed. 

35 [0194] For example, the following two methods are 
considered to change information frames of data control 
information. First, to describe a method for changing 
information frames of data control information in data, 
the default identifier (to be written in a fixed region or 

40 position) of the information present in the data 
described for the information frame of data control infor- 
mation is set and then, information frame change con- 
tents are described. 

[0195] To change information frames of data control 
45 information by describing a method for changing only 
the information frames of data in the control information 
(information frame control information) as another 
metiiod, a default identifier provided for control informa- 
tion is set, the contents of the Information frames of the 
so data control information to be changed are described, 
and ft is comnhunicated to a receiving terminal in 
accordance with ACK/Reject and confirmed that the 
information frames of the data control information are 
changed and thereafter, the data in which information 
65 frames are changed is ti^nsnvtted. Information frames 
of transmission control information and control informa- 
tion can be also changed in accordance with the above 
two methods (Figure 19). 
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[0196] More specifically, though the header informa- 
tion of MPEGS is fixed, by providing a default identifier 
for a progFam map table (defined by PSI) for relating the 
video stream of MPEG2rTs (transport stream) with the 
audio stream of it and defining a configuration stream in 
which a method for changing frames of the information 
for the video stream and audio stream is described, it is 
possible to first interpret the configuration stream and 
then, interpret the headers of the video and audio 
streams in accordance with the conterrt of the configura- 
tion stream when the default identifier is set It is possi- 
ble for the configuration stream to have the contents 
shown in Figure 19, 

[0197] The contents (transmitted-format information) 
of the present invention akx>ut a transmission method 
arxl/6r a structure of the data to be transmitted corre- 
spond to. for example, an information frame in the case 
of the above emtxxiliment. 

[0198] Moreover, for the above emkx)diments, a case 
of transmitting the contents to be changed concerned 
with a transmission method and/or the structure of the 
data to be transmitted is mainly described. However, it is 
also possitde to use a structure for trar^mitting only the 
identifier for the contents. In this case, as shoyvn in Fig- 
ure 44, it is also possible to use an audio^video transmit- 
ter provided with (1) transmitting nieans 5001 for 
transmitting the content concerned with a transmission 
method and/or the structure of the data to be transmit- 
ted or an identifier showing the content as the transmit- 
ted-format informatfon through the transmissfon line 
same as that of the data to be transmitted or a transrnis- 
sion line different from the former transmission line and 
(2) storing means 5002 for storing a plurality of types of 
the contents concerned with the transmission method 
and/or the structure of the data to t>e transmitted and a 
plurali^ of types of identifiers for the contents, in which 
the identifiers are included in at least one of the data 
control informatipn. transmission control information, 
and information for controlling terminal-side processing. 
Moreover, as shown in Rgure 45. it is possible to use an 
audio-video receiver provided with receiving means 
5101 for receiving the transmission format information 
transmitted from the audio-video transmitter and trans- 
mission information interpreting means 5102 for inter- 
preting the received transmission format information. 
Furthermore, the audio-video receiver can be consti- 
tuted with storing means 5103 for storing a plurality of 
types of contents concemed with the transmission 
method and/or the structure of the data to be transmit- 
ted and a plurality of types of identifiers for the contents 
4 Q tsse the contents stored in the storing means to inter- 
pret the contents of the identifiers when receiving the 
identifiers as the transmission format information. 
[0199] More specifically, by preparing a plurality of 
types of information frames previously determined 
between transmitting and receiving terminals and trans- 
mitting identifiers for the above information frames and 
information frame identifiers for a plurality of types of 



data control information, a plurality of types of transmis- 
sion control information, and a plurEdity of types ctf con- 
trol information (information-frame control information) 
together with data or as control information, it is possi- 

5 tAe to identify a plurality of typ^ of data control inforrha- 
tion. a plurality of types of transmissfon control 
information, arxi a plurality of types of control informa- 
tion and optionally select the information frame of each 
type of information in accordance with the type of a 

10 medium to be transmitted or the size of a transmission 
line. Identifiers of the present invention correspond to 
the above informatfon frame identifiers. 
[0200] It is possible to read and interpret these infor- 
mation identifiers and default identifiers even if informa- 

IS tion frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-length 
region or predetermined position of the information to 
be transmitted. 

[0201] Moreover, in addition to the structures 
20 descrit)ed for the above embodiments, it is possible to 
use a structure for temporarily selecting the caption pic- 
ture of a program to be looked and listened by the user 
and showing it for the user when it takes a lot of time to 
set up a necessary program or data by using a broad- 
25 casting channel for broadcasting only the heading pic- 
tures of pictures broadcasted through a plurality of 
channels and switching programs to be looked and lis- 
tened by the user. 

[0202] As described above, the present invention 

30 makes it possfole to change frames of the information 
corresponding to the situation in accordance with the 
purpose or transmission line by dynamically determin- 
ing the frame of data control information, transmission 
control information, or control information used by trans- 

35 mitting and receiving terminals. 

[0203] Moreover, it is possltrie to know whether infor- 
mation frames are changed by providing a default iden- 
tifier for showing whether to transit or receive and 
process information by a publicly-known information 

40 frame previously set between transmitting and receiving 
terminals for data control information, transmission con- 
trol information, arKi control information respectively 
and it is possible to prevent unnecessary configuration 
information from being transmitted even if information 

45 frames of information are not changed by setting a 
default identifier and communicating changed contents 
only when change is performed. 
[0204] Furthermore, it is possible to identify a plurality 
of types of data control information, a plursdity of types 

so of transmission control information. arxJ a plurality of 

inforrnation frames previously determined t)etween 
transmitting and receiving terminals and transmitting 
information frame identifiers for identifying a plurality of 
S5 types of data control infonmation. a plurality of types of 
transmission control information, ard a plurality of types 
of control information together with data or as control 
information and optionally select the information frame 
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of each type of information in accordance with the type 
of a medium to be transmitted or the size of a transmis- 
sion line. 

[0205] It is possible to read and interpret these infor- 
mation identifiers and default identifiers even if informa- s 
Won frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-length 
. region or predetermined position of the information to 
be transmitted. 

[0206] Embodiments of the present invention are io 
descril>ed below by referring to the aocompanying draw- 
ings. 

[0207] In this case, any one of the above-described 
problems (B1) to (B3) is solved. 

[0208] A "picture ( or video)** used for the present is 
invention includes both a static picture and a moving 
picture. Moreover, a purposed picture can be a two- 
dimensional picture such as a computer graphics (CG) 
or threenjimensional picture data constituted with a 
wire-frame model. 20 
[0209] Rgure 25 is a schematic t^lock diagram of the 
picture encoder and a picture decoder of an embodi- 
ment of the present invention. 

[0210] A transmission control section 401 1 for trans- 
mitting or recording various pieces of encoded informa- 25 
tion is means for transmitting the information for coaxial 
cable, CATV, LAN, or modem. A picture encoder 4101 
has a picture encoding section 4012 for encoding pic- 
ture information such as H.263, MPEG1/2, JPEG, or 
IHuffman encoding arxi the transmission control section 30 
401 1 . Moreover, a picture decoder 4102 has an output 
section 401 6 constituted with a reception control section 
4013 for receiving various pieces of encoded Informa- 
tion, a picture decoding section 4014 for decoding vari- 
ous pieces of received picture information, a picture 35 
synthesizing section 401 5 for synthesizing one decoded 
picture or more, and an output section 4016 constituted 
with a display and a printer for outputting pictures. 
[021 1] Rgure 26 is a schematic block diagram of the 
audio encoder and an audio decoder of an en^>odiment 40 
of the present invention. 

[0212] An audio encoder(sound encorder) 4201 is 
constituted with a transmission control section 4021 for 
transmitting or recording various pieces of encoded 
information and an audio encoding section 4022 for 45 
encoding such audio information such as G.721 or 
MPEG1 audio. Moreover, an audio decoder(a sound 
decoder) 4202 Is constituted with a reception control 
section 4023 for receiving various pieces of encoded 
irrformation, an audio decoding section 4024 for decod- so 
ing the above pieces of audio information, an audio syn- 
thesizing section (a sound synthesizing section)4025 
for synthesizing one decoded audio or more, and output 
means 4026 for outputting audio. 

[021 3] Time-series data for audio or picture is specif i- ss 
cally encoded or decoded by the above encoder or 
decoder. 

[0214] The communication environments in Figures 



25 and 26 can be a communication environment in 
which a plurality of logical transmission lines can be 
used without considering muttipiexing means like the 
case of internet or a communication environment in 
which multiplexing means must be conskJered like the 
case of an analog telephone or satellite broadcasting. 
Moreover, a system for bilaterally transferring a picture 
or audio between terminals like a video telephone or 
vkJeo conference or a system for broadcasting a kxoad- 
casting-type picture or audio on satellite broadcasting, 
CATV, or internet is listed as a terminal connection sys- 
tem. 

[0215] Moreover, a method for synthesizing a picture 
arxJ audio can be defined by describing a picture and an 
audio, structural information for a picture and an audio 
(display position and display time), an audio-video 
grouping method, a picture display layer (depth), and an 
object ID (ID for identifying each object such as a pic- 
ture or audio) and the relation between the attritxites of 
them with a script language such as JAVA. VRML, or 
MHEG. A script descril»ng a synthesizing method is 
ok3tained from a network or local memory. 
[021 6] Moreover, it is possble to constitute a transmit- 
ting or receiving terminal by optionally combining an 
optional number of picture encoders, picture decoders, 
audio encoders, and audio decoders. 
[0217] Figure 27(a) is an illustration for explaining a 
priority adding section and a priority deckling section for 
controlling the priority for processing under overioad. A 
priority adding section 31 for deckjing the priority for 
processing encoded information under overioad in 
accordance with a predetermined criteria by an encod- 
ing method such as H.263 or G.723 and relating the 
encoded information to the decided priority is provided 
for the picture encoder 4101 and audio encoder 4201. 
[0218] The criteria for adding a priority are scene 
change in the case of a picture and audio and audioless 
t3locks in the case of a pk:ture frame, stream, or audio 
designated by an editor or user. 
[021 9] A method for addi ng a priority to a communica- 
tion header and a method for embedding a priority in the 
header of a bit stream to be encoded of a video or audk> 
under encoding are considered as priority adding meth- 
ods for defining a priority under overioad. The former 
method makes it possible to obrtain the information con- 
cerned with & priority without decoding the information 
and the latter method makes it possible to independ- 
ently handle a single bit stream witiiout depending on a 
system. 

[0220] As shown in Figure 27(b), when priority infor- 
mation is added to a communk»tion header and one 
picture frame (e.g. intra-frame encoded 1-frame or inter- 
frame encoded P- or B-frame) is divided into a plurality 
of transmission packets, a priority is added only to a 
communication header for transmitting the head of a 
picture frame accessit>ie as single information in tiie 
case of a picture (when priorities are equal in the same 
picture frame, it is possible to assume that the priorities 
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are not changed until the head of the next accessible 
picture frame appears). 

[0221] Moreover, in the case of a decoder, a priority 
deciding section 32 for deciding a processing method is 
provided for the picture decoder 4102 and audio s 
decoder 4202 in accordance with the priorities of vari- 
ous pieces of encoded information received under over- 
load. 

[0222] Rgures 28(a) to 28(c) are illustrations for 
explaining the grading for adding a priority. Decoding is io 
performed by using two types of priorities for deciding 
the priority for processing under overload at a terminal. 
[0223] That is. a stream priority (Stream Priority; inter- 
time-series-data priority) for defining the priority for 
processing under overload In bit streams such as pic- is 
ture and audio and a frame priority (Frame Priority; 
intra-time-series-data priority) for defining the priority for 
processing under overioad In frames such as picture 
frames in the same stream are defined (see Figure 
28(a)). 20 
[0224] The former stream priority makes it possible to 
handle a plurality of videos or audios. The latter frame 
priority makes it possftDle to add a different priority to a 
picture scene change or the same intra-frame encoded 
picture frame (l-frame) in accordance with the intention 25 
of an editor. 

[0225] A value expressed by the stream priority repre- 
sents a case of handling it as a relative value and a case 
of handling it as an absolute value (see Figures 28(b) 
and 28(c)). 30 
[0226] The stream and frame priorities are handled by 
a repeating terminal such as a router or gateway on a 
network and by transmitting and receiving terminals in 
the case of a terminal. 

[0227] 1wo types of methods for expressing an abso- 35 
lute value or relative value are considered. One of them 
is the method shown in Figure 28(b) and the other of 
them is the method shown in Figure 28(c). 
[0228] In Rgure 28(b), the priority of an absolute value 
is a value showing the sequence in. which picture 40 
streams (video streams) or audio streams added by an 
editor or mechanically added are processed (or to be 
processed) under overload (but not a value considering 
the load fluctuation of an actual network or terminal). 
The priority of a relative value is a value for changing the 45 
value of an absolute priority in accordance with the k)ad 
of a t^minal or network. 

[0229] By dividing a priority into a relative value and 
an at>solute value to control the values and thereby 
changing only relative values at the transmitting side or so 
by a repeater in acccrdance with the load fluctuation of 
a network or the like, it is possible to record the value of 
an absolute value into a hard disk or VTR while leaving 
the at^olute priority added to a video or audio stream. 
Thus, when the value of the absolute priority is ss 
recorded, it is possible to reproduce a picture or audio 
that is not influenced by the load fluctuation of a network 
or the Ilka Moreover, it is possitsle to transmit a relative 



or absolute priority through a control channel independ- 
ently of data. 

[0230] Moreover, in Figure 28(b), it is possible to fine 
the grading compared to a stream priority and harxlle a 
frame priority for defining the priority for frsmie process- 
ing under overioad as the value of a relative priority or 
handle it as the value of an absolute priority. For exam- 
ple, by describing an absolute frame priority in encoded 
picture information and describing a relative frame prior- 
ity coaesponding to the absolute priority added to the 
picture frarne in the communication header of a commu- 
nication packet for transmitting encoded Information in 
order to reflect the load fluctuation of a network or termi- 
nal, it is possible to add a priority corresponding to the 
toad of a network or terminal even at a frame level while 
leaving an original priority. 

[0231] Moreover, it is possit)le to transmit a relative 
priority by descn'bing the relation with a frame not in a 
communication header but in a control channel inde- 
pendently of data. Theretiy. it is possible to record data 
into a hard disk or VTR while leaving an absolute priority 
originally added to a picture or aidio stream. 
[0232] Furthermore, in Figure 28(b), when reproduc- 
ing data at a receiving terminal while transmitting the 
data through a network without recording the data at the 
receiving terminal, it is possible to compute the value of 
an absolute priority and that of a relative priority at 
frame and stream levels at the transmitting skie and 
thereafter transmit only ak)solute values because it is 
unnecessary to control absolute and relative values by 
separating them from each other at a receiving terminal. 
[0233] In Figure 28(c), the priority df an absolute value 
is a value uniqu^y determined between frames 
obtained from the relation between Stream Priority and 
Frame Priority. The priority of a relative value Is a value 
showing the sequence in which picture streams or audio 
streams added by an editor or mechanically added are 
processed (or to be processed) under overioad. In the 
case of the example in Figure 28(c), the frame priority of 
a picture or audio stream (relative; relative value) and 
the stream priority for each stream are added. 
[0234] An absolute frame priority (absolute; absolute 
value) is obtained from the sum of a relative frame prior- 
ity and a stream priority (That is. absolute frame priority 
= relative frame priority -i- stream priority). To obtain an 
at)solute frame priority, it is also possS>le to use a sub- 
tracting method or a constant-multiplying method. 
[0235] An absolute frame priority mainly uses a net- 
work. This is because the expression using an absolute 
value does not require the necessity for deciding a pri- 
ority for each frame through a repeater such as a router 
or gateway by considering Stream Priority and Frame 
Priority. using the absolute frame priority, such 
processing as disuse of a frame by a repeater is simpli- 
fied. 

[0236] Moreover, it can be expected to apply a relative 
frame priority mainly to an accumulation system for per- 
forming recording or editing. In the case of an editing 
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operation, a plurality of picture and audio streams niay 
be handled at the same time. In this case, flie number of 
picture streams or the number of frames that can be 
reproduced may be limited depending on the IockJ of a 
terminal or network. 5 
[0237] In the above case, rt is unnecessary to recalcu- 
late every Frame Priority differentiy from the case in 
which an absolute value is expressed only by separat- 
ing Stream Priority from Frame Priority, that is, only by 
changing Stream Priority of a stream which an editor io 
wants to preferentially display or a user wants to see. 
Thus, it is necessary to use an absolute expression or a 
relative expression in accordance with the purpose. 
[0238] By describing whettier to use a stream priority 
as a relative value or absolute value, ft is possible to is 
effectively express a priority for transmission and accu- 
mulation. 

[0239] In the case of the example in Figure 28(b). tt is 
differentiated by following a stream priority that the 
value expressed by tiie stream priority is a relative value 20 
or absolute value by using a flag or identifier for 
expressing whether the value expressed by the stream 
priority is an absolute value or relative value. In the case 
of a frame priority, a flag or identifier is unnecessary 
because a relative value is descrio^ed in a communica- ss 
tion header and an absolute value is described in an 
encoded frame. 

[0240] In the case of the example in Figure 28(c). a 
flag or identifier for identifying whether a frame priority is 
an absolute value or relative value is used. In the case 30 
of an absolute value, the frame priority is a priority cal- 
culated in accordance with a stream priority and a rela- 
tive frame priority and therefore, tiie calculation is not 
performed by a repeater or terminal. Moreover, when 
the calculation formula is already known at a terminal, it ss 
is possible to inversely calculate a relative frame priority 
from an absolute frame priority and a stream priority. 
For example, it is also possible to obtain the absolute 
priority (Access Unit Priority) of a packet to be transmit- 
ted from the relational expressfon 40 
[0241] "Access Unit Priority = sta-eam priority - frame 
priority". 

In this case, it is also possible to express the frame pri- 
ority as a degradation prtority because it is obtained 
after being subtracted from the sfream priority. 45 
[0242] Moreover, it is also possible to control data 
processing by relating one stream priority or more to the 
priority for processing of the data passing through the 
logical channel of TCP/IP {port Na of LAN). 
[0243] Furtiiermore. it is expected tiiat the necessity so 
for retransmission can be reduced by assigning a 
sfream priority or frame priority lower than ttiat of a 
character or control information to a picture or audio. 
This is because no problem occurs in most cases even 
if a part of a picture or audio is lost. ss 
[0244] Figure 29 is an illusfration for explaining a 
method for assigning a priority to multi-resolution video 
data. 



[0245] When one sfream is constituted witii a plurality 
of sut>sfreams . it is possible to d^ine a substream 
processing method by adding a stream priority to the 
subsfreams and describing a logical sum or logical 
product under accumulation or transmission. 
[0246] In ttie case of a wavelet, it is possible to decom- 
pose one picture frame Into a plurality of different-reso- 
lution picture frames. Moreover, even in the case of a 
DCT-base encoding method, it is possible to decom- 
pose one picture frame into a plurality of different-reso- 
lution picture frames by dividing the picture frame into a 
high-frequency component and a low-frequency com- 
ponent and encoding them. 

[0247] In addition to stream priorities added to a plu- 
rality of picture sfreanis constituted with a series of 
deconposed pictijre frames, tfie relation between pic- 
ture sfreams is defined witti AND (logical product) and 
OR (logical sum) in order to describe the relation. Spe- 
cifically, when the stream priority of ia sfream A is 5 and 
that of a sfream B Is 10 (the smaller a numerical value 
gets, the higher a priority t>ecomes). the relation 
between pictijre sfreams is defined ttiat the sfream B is 
disused in tiie case of disuse of sfream data depending 
on the priority but the sfream B is transmitted and proc- 
essed without being disused even If the priority of the 
stream B is lower ttian ttie priority of a threshold in the 
case of AND by describing the relation between 
streams. 

[0248] Thereby, relevant sfreams can be processed 
wtthout being disused. In ttie case of OR. it is defined 
that relevant streams can be disused. It is possible to 
perform disuse processing at a fransmitting or receiving 
terminal or a repeating terminal as ever. 
[0249] Moreover, when the same video dip is 
encoded to 24 Kbps and 48 Kbps respectively as an 
operator for relational description, there is a case in 
which either 24 or 48 Kbps nnay be reproduced (exclu- 
sive logical sum EXOR as relational description). 
[0250] When tiie priority of tiie former is set to 1 0 and 
that of the latter is set to 5, a user can reproduce ttie lat- 
ter in accordance with a priority or select the latter with- 
out following the priority. 

[0251 ] Figure 30 is an illustration for explaining a com- 
munication payload constituting metiiod. 
[0252] When constituted with a plurality of sut>- 
streams. disuse at a fransmission packet level becomes 
easy by, for example, constituting fransmission packets 
starting with, for example, one having the highest prior- 
ity in accordance with a stream prfority added to a sut>- 
stream. Moreover, disuse at a communication packet 
level becomes easy by fining grading and uniting ttie 
information for objects respectively having a high frame 
prfority and thereby constituting a communication 
packet. 

[0253] By relating the sliced sfructure of a picture to a 
communication packet, return of a missing packet 
becomes easy. That is. by relating the sliced structure of 
a video to a packet sfructure, a re-sync marker for 
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resynchronization is unnecessary. Unless a sliced 
structure coincides with the stnicture of a communica- 
tion packet, it is necessary to add a re-sync marker 
(marker for making a returning position known) so that 
resynchronization can be performed if information is 5 
damaged due to a missing packet). 
[0254] In accordance with the above-mentioned, it is 
considered to apply a high error protection to a commu- 
nication packet having a high priority Moreover, the 
sliced structure of a picture represents the unit of col- 10 
lected picture information such as GOB or MB. 
[0255] Rgure 31 is an illustration for explaining a 
method for relating data to communication payload. By 
transmitting a method for relating a stream or object to a. 
conrvnunication packet togetiier with control information is 
or data, it is possHsle to generate an optional data format 
in accordance with the communication state or purpose. 
For example, in the case of RTP (Real time Transfer 
Protocol), the payioad of RTP is defined for each encod- 
ing to be handled. The fonnat of the existing RTP is 
fixed. In the case of H.263, as shown in Figure 31. three 
data formats from Mode A to Mode C are defined. In the 
case of H.263. a communication payload purposing a 
multi-resolution picture format is not defined. 
[0256] In the case of the example in Rgure 31. Layer 
No. and the above relational description (AND. OR) are 
added to the data format of Mode A and defined. 
[0257] Rgure 32 is an illustration for explaining the 
relation between frame priority, stream priority, and 
communication packet priority. 

[0258] Moreover. Rgure 32 shows an example of 
using a priority added to a communication packet on a 
transmission line as a communication packet priority 
and relating a stream priority and a frame priority to the 
communication packet priority. 

[0259] Generally, in the case of communication using 
IP, it is necessary to transmit data by relating a frame 
priority or stream priority added to picture or audio data 
to the priority of a low-order IP packet. Because the pic- 
ture or audfo data is divided into IP packets and trans- 
mitted, it is necessary to relate priorities to each other. 
In the case of the example in Rgure 32. because the 
stream priority takes values from 0 to 3 and the frame 
priority tal^ values from 0 to 5. high-order data can 
take priorities from 0 to 1 5. 

[0260] In the case of IPv6, priorities (4 bits) from 0 to 
7 are reserved tor congestion-controlled traffic. Priori- 
ties from 8 to 15 are reserved for real-time comnunica- 
tion traffic or not-congestion-controlled traffic. Priority 
15 is the highest priority and priority 8 is tiie lowest pri- 
ority. This represents the priority at the pa<^6t level of IP. 
[0261] In the case of data transmission using IP, it is 
necessary to relate high-order priorities from 0 to 15 to 
low-order IP priorities from 8 to 1 5. To relate priorities to 
each other, it is possible to use a method of clipping 
some of high-order priorities or relate priorities to each 
otiier by using a performance function. Relating of high- 
order data with a- low-order IP priority is performed at a 



repeating node (router or gateway) or transmitting and 
receiving terminals. 

[0262] Transmitting means is not restricted to only IP. 
It is possik)le to use a transmission packet having a flag 
showing whether rt can be disused like TS (transport 
stream) of ATM or MPEG2, 

[0263] The frame priority and stream priority having 
been described so far can be applied to a transmitting 
medium or data-recording medium. It is possible to use 
a floppy diisk or optical disk as a data-recording, 
medium. 

[0264] Moreover, it is possible to use not only the 
floppy disk or optical disk but also a medium such as an 
IC card or ROM cassette as long as a program can be 
recorded in the medium. Furthermore, it is possible to 
use an audio-video repeater such as a router or gate- 
way for relaying data. 

[0265] Furthermore, preferential retransmission is 
realized by deckling time-series data to be retransmit- 
ted in accordance witii the information of Stream Prior- 
ity (inter-time-series-data priority) or Frame Priority 
Cintra-time-series-data priority). For exanple. when 
decoding is performed at a receiving terminal in accord- 
ance with priority information, it is possSsle to prevent a 
stream or frame that is not an object for processing from 
being retransmitted. 

[0266] Furthermore, separately from a present priority 
to be processed, it is possible to deckle a stream or 
frame having a priority to ft>e retransmitted in acoord- 
arKe with the relation between retransmission fre- 
quency and successful transmission frequency 
[0267] Furthermore, in the case of a transmitting-side 
terminal, preferential transmission is realized by decid- 
ing time-series data to be transmitted in accordance 
with the information of Stream Priority (inter-time- 
series-data priority) or Frame Priority (intra-time-series- 
data priority). For example, by deciding the priority of a 
stream or frame to be transmitted in accordance with an 
average transfer rate or retransmission frequency, it is 
possfole to transmit an adaptive picture or audio even 
when a network is overioaded. 

[0268] The above embodiment is not restricted to two- 
dimensionali3icture synthesis. It is also possible to use 
an expression method obtained by combining a two- 
dimensional picture with a three^imensional picture or 
include a picture-syritiiesizing method for synthesizing 
a plurality of pictures so as to be adjacent to each other 
a wide-visual-fieM picture (panorama picture). 
Moreover, communication systems ptarposed by ttie 
present Invention are not restricted to bkJirectional 

\jr\t V SJt O'ts^^i^. I XJt C^CI,II^I9, UCll»0«IU99IWII wi piwuiioo 

and audio from a center-side terminal to a house-skle 
terminal can use radio waves (e.g. VHP band or UHF 
band) or satellite broadcasting and information origina- 
tion from the house-side terminal to the center-side ter- 
minal can use an arialog telephone line or N-ISDN (it is 
not always necessary that pictures, audio, or data are 
multiplexed). Moreover, it is possible to use a communi- 



25 



30 



35 



40 



45 



50 



24 



47 



EP 0 905 976 A1 



48 



cation system using radio such as an IrDA. PHS (Per- 
sonal Handy Phone) or radio LAN. 
[0269] Furthermore, a purpose terminal can be a port- 
able terminal such as a portable information terminal or 
a desktop terminal such as a set-top BOX or personal s 
computer, 

[0270] As described above, the present Invention 
. makes it easy to handle a plurality of video streams and 
a plurality of audio streams and mainly synchronise and 
reproduce important scene cut together with audio by io 
reflecting the intention of an editor 
[0271] An embodiment of the present invention is 
desaibed below by referring to the accompanying draw- . 
Ings. 

[0272] The emtxxjiment described below solves any is 
one of the above problems (C1) to (C3). 
[0273] Figure 33 shows the structure of the transmitter 
of the first embodiment. Syrr^l 2101 denotes a picture- 
irput terminal and the size of a sheet of picture has 144 
pixels by 176 pixels. Symbol 2102 denotes a video 20 
encoder that is constituted with four components 1021, 
1022, 1023. and 1024 (see Recommendation H.261): 
[0274] Symbol 1021 denotes a switching unit for divid- 
ing an input picture into macrot^locks (a square region of 
16 pixels by 16 pixels) and deciding whether to intra- 25 
encode or inter-encode the blocks and 1022 denotes 
movement compensating means for generating a move- 
ment compensating picture in accordance with the local 
decoded picture which can be calculated in accordance 
witii the last-time encoding result, calculating the differ- so 
ence between the movement compensating picture and 
an input picture, and outputting the result in macrob- 
locks. Movement compensation includes haKpixel pre- 
diction tiaving a long processing time and fullpixel 
prediction having a short processing time. Symtx)l 1 023 3S 
denotes orthogonal transforming means for applying 
DOT transformation to each macroblock and 1024 
denotes variabie-iength-encoding means for applying 
entropy encoding to the DCT transformation result aiKl 
other encoded information. 40 
[0275] Symbol 2103 denotes counting means for 
counting execution frequencies of four components of 
the video encoder 2102 and outputting the counting 
result to transforming means every input picture. In this 
case, the execution frequency of the halfpixel prediction 4s 
and that of the fullpixel prediction are counted from the 
movement compensating means 1 022. 
[0276] Symbol 2104 denotes transforming means for 
outputting the data string shown in Figure 34. Symtx)l 
21 05 denotes transmitting means for multiplexing a var- so 
lable-length code sent from tiie video encoder 2102 and 
a data string sent from the transforming means 2104 
into a data string and outputting the data string to a data 
output terminal 2109. 

[0277] According to ttie at>ove structure, it is possible ss 
to transmit the execution frequencies of indispensak^le 
processing (switching unit 1021, orthogonal transform- 
ing means 1023, and variable-length encoding means 



1 024) and dispensable processing (movement compen- 
sating means 1 022) to a receiver. 
[0278] The transmitter of the first embodiment con^e- 
sponds to claim 68. 

[0279] Figure 40 is a flowchart of the transmitting 
metiiod of tiie second emtxxliment. 
[0280] Because operations of this embodiment are 
similar to ttiose of the first embodiment, corresponding 
elements are added. A picture is input in step 801 (pic- 
ture input terminal 2101) and the picture is divkied into 
macrok}locks in step 802. Hereafter, processings from 
step 803 to step 806 are repeated until the processing 
corresponding to every macrobtock is conrpleted in 
accordance with the conditional branch in step 807. 
Moreover, when each processing is executed so that 
frequendes of the processings from step 803 to step 
806 can be recorded in specific variat)les. a correspond- 
ing variat)le is incremented by 1 . 
[0281] First it is decided whether to intra-encode or 
inter-encode a macrot)lock to be processed in step 803 
(switching unit 1021). When inter-encoding the macrob- 
locK movement compensation Is performed in step 804 
(movement compensating means 1022). Thereafter, 
DCT transformation and variedDle-iength encoding are 
performed in steps 805 arxi 806 (ortiiogonal trEmsformr 
ing means 1023 and variable-length encoding means 
1024 ). When processing for every macrotslock is com- 
pleted (in the case of Yes in step 807), tiie variat>ie 
showing the execution frequency corresponding to each 
processing is read in step 808, the data string shown in 
Rgure 2 is generated, and the data string and a code 
are multiplexed and output The processings from step 
801 to step 808 are repeatedly executed as long as 
input pictures are continued. 

[0282] The above structure makes it possible to trans- 
mit the execution frequency of each processing. 
[0283] The transmitting method of the second emkxxi- 
iment corresponds to claim 67. 
[0284] Figure 35 shows the structure of the receiver of 
the third emtxxliment. 

[0285] In Rgure 35, symbol 307 denotes an input ter- 
minal for inputting the output of the transmitter of tiie 
first emtK)diment and 301 denotes receiving means tor 
fetching a variable-length code and a data string 
tiirough inverse multiplexing in accordance with tiie out- 
put of the transmitter of the first embodiment and out- 
putting tiiem. In this case, it is assumed that the time 
required to receive the data for one sheet is measured 
arxi also output. 

[0286] Symtxsl 303 denotes a decoder for a video 
using a variable-length code as an input, which is con- 
stituted with five components. Symbol 3031 denotes 
variable-length decoding means for fetching a DCT 
coefficient and other encoded information from a varia- 
ble-length code, 3032 denotes inverse orthogonal trans- 
forming means for applying inverse DCT transformation 
to a DCT coefficient, and 3033 denotes a switching unit 
for switching an output to upskJe or downside every 
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macroblock in accordance with the encoded information 
showing whether the macrdt)lock is intra-encoded or 
inter-encoded. Symtxjl 3034 denotes movement com- 
pensating means for generating a movement compen- 
sating picture by using the last-time decoded picture 5 
and movement encoded informatioa and adding and 
outputting the outputs of the inverse orthogonal trans- 
forming means 3032. Symbol 3035 denotes execution- 
time measuring means for measuring and outputting the 
execution time until decoding and outputting of a picture 10 
is complied after a variable-length code is input to the 
decoder 303. 

[0287] Symbol 302 d^otes estimating means for 
receiving the execution frequency of each element (var- 
iable-length decoding means 3031 . inverse orthogonal is 
transforming means 3032. switching unit 3033, or 
movement compensating means 3034) from a data 
string sent from the receiving means 301 arxJ execution 
time from the execution-time measuring means 3035 to 
estimate the execution time of each element. 20 
[0288] To estimate the execution time of each ele- 
ment, it is possitde to use the linear regression and 
assume an estimated ^ecution time as a purposed var- 
iable y and the execution frequency of each component 
as an explanatory variable xui. In this case, it may be 25 
possitDle to regard a regression parameter avi as the 
execution time of each element Moreover, in the case 
of linear regression, it is necessary to accumulate 
much-enough past data and resultantly. many memo- 
ries are wasted. However, to avoid many memories from 30 
being wasted, it is also possit)le to use the estimation of 
an internal-state variat>le by a Kalman filter. It is possi- 
YAe to consider the akxTve case as a case in which an 
ot>served value is assumed as an execution time, tiie 
execution time of each element is assumed as an Inter- 3S 
nal-state variable, and an observation matrix C changes 
every step due to tiie execution frequency of each ele- 
ment. Symbol 304 denotes frequency reducing means 
for changing the execution frequency of each element 
so as to reduce the execution frequency of fullpixel pre- 40 
diction and increase the execution frequency of halfpixel 
prediction by a conresponding value. The method for 
calculating the corresponding value is shown below. 
[0289] Rrst the execution frequency and estimated 
execution time of each element are received from the 45 
estimating means 302 to estimate an execution time. 
When the execution time exceeds the time required to 
receive the data from the receiving means 301 . the exe- 
cution frequency of fullpixel prediction is increased and 
the execution frequency of halfjpixel precfiction is so 
decreased until the former time does not exceed the lat- 
ter time. Symtxd 306 denotes an output terminal for a 
decoded picture.. 

[0290] Moreover, there is a case in which the move- 
ment compensating means 3034 is designated so as to ss 
perform halfpixel ((prediction in accordance with encoded 
information. In this case, when the predetermined exe- 
cution frequency of halfpixel prediction is exceeded, a 



halfpixel movement is rounded to a ful^i^l movement 
to execute fullpixel prediction. 

[0291 ] According to tiie aljove-described first and third 
errfoodiments. the execution time of decoding is esti- 
mated in accordance with the estimated execution time 
of each element and. when the decoding execution time 
may exceed the time (designated time) required to 
receive the data for one sheet, hal^ixel prediction hav- 
ing a long execution time is replaced with fullpi^ pre- 
diction. Theret3y. it is possfole to prevent an execution 
time from exceeding a designated time and solve the 
protDlem (CI) (corresponding to claims 68 and 74). 
[0292] Moreover, a case of regarding the parts of 
indispensable and dispensable processings as two 
groups corresponds to claims 66 and 72 and a case of 
regarding the part of a video as waveform data corre- 
sponds to claims 64 and 70. 

[0293] Furthermore, by using no high^requency com- 
ponents in the IDCT calculation by a receiver, it is pos- 
sible to reduce the processing time for tiie IDCT 
calculation. That is, by regarding tiie calculation of low- 
frequency components as indispensak>le processing 
and the calculation of hugh-frequency components as 
dispensable processing in the IDCT calculation, it is 
also possible to reduce the calculation frequency of 
high-frequency conrponents in the IDCT calculation. 
[0294] Figure 41 is a flowchart of the receiving method 
of the fourtii emtxxfimertt 

[0295] Because operations of this emfc>odiment are 
^milar to those of the third ennbodiment corresponding 
elements are added. In step 901, the variat^e a J for 
expressing the execution time of each element is initial- 
ized (estimating means 302) . in step 902, multipleiced 
data is input and the time required for multiplexing the 
data is measured (receiving means 301) . In step 903, 
the rrujltiplexed data is divided into a variak^le-length 
code and a data string and output (receiving means 
301). In step 904. each execution frequency is fetched 
from a data string (Figure 2) and it is set to xj. In step 
905. an actual execution frequency is calculated in 
accordance with the execution time aj of each element 
and each execution frequency xJ (frequency reducing 
means 304). In step 906, measurement of the execution 
time for decoding started. In step 907, a decoding 
routine to be described later is started. Thereafter, in 
step 908. measurement of the decoding execution time 
is ended (video decoder 303 and execution-time meas- 
uring means 3035). In step 908. the execution time of 
each element is estimated in accordance with the 
decoding execution time in step 908 and the actual exe- 
cution frequency of each eiemem in stqD 905 to update 
aJ (estimating means 302). The above processing is 
executed every input multiplexed data 
[0296] Moreover, in step 907 for decoding routine, var- 
iable-length decoding is performed in step 910 (varia- 
ble-length decoding means 3031). inverse orthogonal 
transformation is performed in step 91 1 (inverse ortiiog- 
onal transforming means 3032). and processing is 



26 



51 



EP0905976A1 



52 



branched in step 91 2 in accordance with the infbrnnation 
of the intra-/inter-processing fetched through the 
processing in step 910 (swvitching unit 3033). In the case 
of inter-processing, inovement conrpensation is per- 
formed in step 913 (movement compensating means 
3034). In step 913, the execution frequency of hatfpixel 
prediction is counted in step 913. When the counted 
execution frequency exceeds the actual execution fre- 
quency obtained in step 905, hatfpixel prediction is 
replaced with f uilpixel prediction for execution. After ttie 
above processing is applied to every macroblock (step 
914), the routine is ended. 

[0297] According to the atxsve-described second and 
fourth emtKxdiments. the execution time of decoding is 
estimated in accordance with the estimated execution 
time of each element and. when the execution time may 
exceed the time required to receive the datai for one 
sheet (designated time), halfpixel prediction having a 
long execution time is replaced with fullpixel prediction. 
Thereby, it is possible to prevent an execution time from 
exceeding a designated time and solve the problem 
(CI) (corresponding to cl€ums 67 and 73); 
[0298] Furthermore, a case of regarding tiie parts of 
dispensable and indispensable processings as two 
groups corresponds to claims 65 and 71 and a case of 
regarding the part of a video as waveform data corre- 
sponds to claims 63 and 69. 

[0299] Rgure 36 shows the structure of tiie receiver of 
the f iftfi embodiment. 

[0300] Most components of this embodiment are tiie 
same as those descrS^ed for the seoorKi embodiment. 
However, two added components and one corrected 
component are descrit>ed below. 
[0301] Symbol 402 denotes estimating means 
otJtained by correcting the estimating means 302 
descrit>ed for the second embodiment so as to output 
the execution time of each element obtained as the 
result of estimation separately from an output to fre- 
quency limiting means 304. Symbol 408 denotes trans- 
mitting means for generating the data string shown in 
Rgure 37 in accordance with the execution time of each 
element and outputting it. When expressing an execu- 
tion time with 16 bits by using microsecond as the unit, 
up to approx. 65 msec can be expressed. Therefore. 
approK. 65 msec will be enough. Symbol 409 denotes 
an output terminal for transmitting the data string to 
transmitting means. 

[0302] Moreover, a receiving method con^esporKling to 
the fifth embodiment can be obtained only tsy adding a 
step for generating the data string shown in Figure 37 
immediately after symbol 808 in Rgure 40. 
[0303] Figure 38 shows tiie sti-ucture of the transmitter 
of the sixth emt>odiment. 

[0304] Most components of this emtxxdiment are the 
same as those described for the first emt>odiment. How- 
ever, two added corrponents are described below. Sym- 
tx>l 606 denotes an input terminal for receiving a data 
string output by the receiver of the third embodiment 



and 607 denotes receiving means for receiving the data 
string and outputting the execution time of each ele- 
ment Symbol 608 denotes deciding means for obtain- 
ing the execution frequency of each element and its 

5 ot>taining procedure is described below. First, every 
macroblock in a picture is processed by the switching 
unit 1021 to obtain the execution frequency of the 
switching unit 1021 at this point of time. Moreover, it is 
possible to uniquely decide execution frequencies by 

10 the movement compensating means 1022. orthogonal 
transforming means 1023. and variable-length encod- 
ing means 1024 in accordance with the processing 
result up to this point of time. Therefore, tiie execution 
time required for decoding at the receiver side is esti- 

15 mated by using these execution frequencies and the 
execution time sent from the receiving means 607. The 
estimated decoding time is obtained as the total sum of 
tiie product k)etween the execution time and execution 
frequency of each element every element Moreover, 

20 when the estimated decoding time is equal to or more 
than the time required to transmit the number of codes 
(e.g. 16 Kbits) to be generated through this picture des- 
ignated by a rate controller or the like (e.g. 250 msec 
when a transmission rate is 64 Kbits/sec), the execution 

25 frequency of fullpixel prediction is increased and the 
execution frequency of haKjpixel prediction is decreased 
so that the estimated decoding execution time does not 
exceed the time required for transmission. (Becaiuse 
fullpixel prediction has a shorter execution time, it is 

30 possible to reduce the execution time of fullpixel predic- 
tion by redudng the frequency of fullpixel prediction.) 
[0305] Moreover, the video encoder 2102 performs 
various processings in accordance with the execution 
frequency designated by the deciding means 608. For 

35 exarrple. after the movement compensating means 
1022 executes halfpisffil predictfon by tiie predetermined 
execution frequency of halfpixel prediction, it executes 
only fullpixel prediction. 

[0306] Furthermore, it is possible to improve the 

40 selecting method so that halfpixel prediction is uniformly 
dispersed in a picture. For example, it is possible to use 
a method of first obtaining every macrotDlock requiring 
halfpixel prediction, calculating the product (3) obtained 
by dividing the number of the above macroblocks (e.g. 

45 12) by the execution frequency of hal4)ixel predfotion 
(e.g 4). and applying halfpixel prediction only to a mac- 
rot>lock whose sequence from the beginning of the mac- 
roblocks requiring halfpixel prediction is divided by the 
akx>ve product without a remainder (0, 3, 6, or 9). 

so [0307] According to the aftxive-described fifth and 
sixth embodiments , the execution time of each esti- 
mated element is transmitted to the transmitting side, 
the execution time of decoding is estimated at the trans- 
mitting side. aruJ halfpixel prediction having a long exe- 

55 cution time is replaced with fullpixel prediction so that 
the estimated decoding execution time does not exceed 
the time (designated time) probably required to receive 
tiie data for one sheet. Thereby, the information for half- 
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pixel prediction among the sent encoded information is 
not disused and theret>y. it is possOile to prevent an exe- 
cution time from exceeding a designated time and solve 
the prolDlem (C2) (corresponding to claims 76 and 78). 
[0308] Moreover, in the case of dispensable process- s 
ing. it is possible to divide inter-macroblock encoding 
into su(^ three movement con^nsations as normal 
movement condensation. 8x8 movement compensa- 
tion, and overlap movement conpensation. 
[0309] Figure 42 is a flowchart of the transmitting io 
method of the seventh emtxxiiment. 
[0310] Because operations of this embodiment are 
similar to those of the sixth en^xxjiment. corresponding 
elements are added. In step 1 001 . the initial value of the 
execution time of each processing is set. A picture is is 
input (input temiinal 2101) in step 801 and it is divided 
into macrc^ocks in st^ 802. In step 1002. it is decided 
whether to intra-encode or inter-encode every macrot>- 
lock (switching unit 1021). Resultantly, the execution 
frequency of each processing from step 1005 to step 20 
806 is known. Therefore, in step 1003. an actual execu- 
tion frequency is calculated in accordance with the 
above execution frequency and the execution time of 
each processing (deciding means 608). 
[0311] Hereafter, the processings from step 1005 to 25 
step 806 are repeated until the processing for every 
macroblock is completed in accordance with the condi- 
tional branch in step 807. 

[0312] Moreover, when each processing is executed, 
a corresponding variable is incremented by 1 so that the 30 
processing frequencies from st^ 1005 to step 806 can 
be recorded in a specific variat^e. First, in step 1005, 
branching is performed in accordance with the dedsion 
result in step 1002 (switching unit 1021). In the case of 
inter-encoding, nrxivement compensation is performed ss 
in step 804 (nxivement compensating means 1022). In 
this case, the frequency of halfpixel prediction is 
counted. When the counted frequency exceeds the 
actual frequency obtained in step 1003. fullpixel predic- 
tion is executed instead without executing halfpixel pre- 40 
diction. Thereafter, in steps 805 and 806, DCT 
transformation and variable-length encoding are per- 
formed (orthogonal transforming means 1023 and vari- 
at)le-length encoding means 1024). When the 
processing for every macroblock is completed, (in the 45 
case of Yes in step 807). the variat)le showing the exe- 
cution frequency corresponding to each processing is 
read in step 808, the data string shown in Figure 2 is 
generated, and the data string and a code are multi- 
plexed and output In step 1004, the data string is so 
reoerved and the execution time of each processing is 
fetched from the data string and set. 
[0313] Processings from step 801 to step 1004 are 
repeatedly executed as long as pictures are input 
[031 4] According to the paragraph beginning with the ss 
final "Moreover" of the desaiptive portion of the fifth 
embodiment and the seventh embodiment, the esti- 
mated execution time of each element is transmitted to 



the transmitting side, the execution time of decoding is 
estimated at the transmitting side, and halfpixel predic- 
tion having a long execution time is replaced with full- 
pixel predictiori so that the estimated decoding 
execution time does not exceed the time (designated 
time) probably required to receive the data for one 
sheet. Thereby, the information for halfpixel prediction 
among the sent encoded information is not disused and 
it is possible to prevent tiie execution time from exceed- 
ing the designated time and solve ttie problem (C2) 
(corresporKling to claims 75 and 77). 
[031 5] Figure 39 shows tiie structure of the transmit- 
ting apparatus of the eighth embodiment of the present 
invention. 

[0316] Most components of this embodiment are the 
same as tiiose described for the first enrtfxxiiment 
Therefore, four added components are described 
below. 

[0317] Symt>ol 7010 denotes execution-time measur- 
ing means for measuring the execution time until encod- 
ing and outputting of a picture are completed after the 
picture is input to an encoder 2102 and outputting the 
measured execution time. Synlkx>l 706 denotes estimat- 
ing means for receiving execution firequenctes of ele- 
ments (switching unit 1021. movement compensating 
means 1022. orthogonal transforming means 1023. and 
variable-lengtii decoding means 1024) of a data string 
from counting means 2103 and the execution time from 
the execution-time measuring means 7010 and estimat- 
ing tiie execution time of each element It is possible to 
use an estimating method same as that described for 
the estimating means 302 of the second enrdxxjiment. 
SynrM 707 denotes an input t^minal for inputting a 
frame rate value sent from a user and 708 denotes 
deciding means for obtaining the execution frequency of 
each element The obtaining procedure is descrfoed 
below. 

[0318] First, every macroblock in a picture is proc- 
essed by the switching unit 1 021 to obtain the execution 
frequency of the switching unit 1 02 1 at this point of time. 
Thereafter, it is possible to uniquely deckie execution 
frequencies by the movement compensating means 
1022. orthogonal transfornting means 1023. and varia- 
fcde-length encoding means 1 024 in accordance with the 
processing result up to this point of time. Then, the total 
sum of products between the execution frequency and 
ttie estimated execution time of each element sent from 
the estimating means 706 is obtained every element to 
calculate an estimated encoding time. Then, when the 
estimated encoding time is equal to or long^ than the 
time usable for encodir^ ot a sheet ot picture obtained 
from the inverse number of the frame rate sent from 
symbol 707, tiie execution frequency of fullpixel predic-. 
tion is increased and that of halfjaixel prediction is 
decreased. 

[0319] By repeating the atxsve change of execution 
frequencies and calculation of the estimated encoding 
time until the estimated encoding time becomes equal 



28 



55 



EP 0 905 976 A1 



56 



to or shorter than the usable time, each execution fre- 
quency is decided. 

[0320] Moreover, the video encoder 2102 performs 
various processings in accordance with the execution 
frequency designated by the deciding means 608. For 
example, after the movement compensating means 
1022 executes halfjpixel prediction by the predetermined 
execution frequency of haHpixel prediction, it executes 
only fullpixel prediction. 

[0321] Furthermore, it is also possible to improve a 
selecting method so that halfpi^l prediction is uniformly 
dispersed in a picture. For example, it is possible to use 
a method of obtaining e/ery macroblock requiring half- 
pixel prediction, calculating the product (3) obtained by 
dividing the number of macrol^locks requiring halff»xel 
prediction (e.g. 12) by the execution frequency of half- 
pixel prediction (e.g. 4). and applying halfplxel predic- 
tion only to a macroblock whose sequence from the 
beginning of the macrot)locks requiring halfpixel predic- 
tion is divided by the product without remainder (0, 3. 6, 
or 9). 

[0322] The above eighth embodiment makes it possi- 
ble to solve the problem (C3) by estimating the execu- 
tion time of each processing, estimating an execution 
time required for encoding in accordance with the esti- 
mated execution time^ and deciding an e)«cution fre- 
quency so that the estimated encoding time becomes 
equal to or shorter than the time usable for encoding of 
a picture determined in accordance with a frame rate 
(corresporxiing to daim 80). 

[0323] Moreover, because the movement compensat- 
ing means 1022 detects a movement vector, there is a 
full-search movement-vector detecting method for 
detecting a vector for minimizing SAD (sum of absolute 
values of differences every pixeO among vectors in a 
range of 15 horizontal and vertical pixels. Furthermore, 
there is a three-step movement-vector detecting 
method (described in annex of H.261). The three-step 
movement-vector detecting method executes the 
processing of selecting nine points uniformly arranged 
in the above retrieval range to select a point having a 
minimum SAD and then, selecting nine points again in a 
narrow range close to the above point to select a point 
having a minimum SAD one more time. 
[0324] It is also possible to property decrease tiie exe- 
cution frequency of the full-search movement-vector 
detecting mettiod and properly increase tiie execution 
^ti^uency of tiie three-step movement-vector detecting 
method by regarding these two methods as a dispensa- 
ble processing metiiod and estimating the execution 
time of each of the two methods, estimating an execu- 
tion time required for encoding in accordance with the 
estimated execution time so that the estimated execu- 
tion time becomes equal to or shorter than the time des- 
ignated by a user. 

[0325] Moreover, it is possible to use a movement- 
vector detecting method using a fixed retrieval fre- 
quency and further simplifying the processing or a 



movement-vector detecting method of returning only 
tiie movement vector (0, 0) as a result together with the 
three-step movement-vector detecting method. 
[0326] Figure 43 is a flowchart of the transmitting 

5 method of the ninth embodiment. 

[0327] Because operations of tills emlxxliment are 
similar to those of the eighth emtxxiiment correspond- 
ing elements are added. For the detailed operation in 
each flow, refer to the description of corresponding ele- 

10 ments. 

[0328] Moreover, because this embodiment is almost 
the same as tfie second embodiment, only different 
points are explained below. 

[0329] In step 11 01 . the Initial value of tiie execution 
IS time of each processing is set to a variable aj. Moreo- 
ver, in step 1102. a frame rate is input (input terminal 
707). In step 1103, an actual execution frequency is 
decided in accordance with the frame rate and the exe- 
cution time aJ of each processing in step 1 102 and the 
20 execution frequency of each processing obtained from 
the intra-Zinter-processing dedsfon result in step 1002 
(deciding means 708). In steps 1 105 and 1106. the exe- 
cution time of encoding is measured. In step 1 104. the 
execution time of each processing Is estimated in 
25 accordance with the execution time obtained in step 
1106 and the actual execution frequency of each 
processing to update tiie variable aJ (estimating means 
706). 

[0330] According to the atDOve-described ninth 
30 embodiment, the execution time of each processing is 
estimated arvJ an execution time required for encoding 
is previously measured in accordance with the esti- 
mated execution time. Thus, it is possit)le to solve the 
problem (C3) by deciding an actual execution frequency 
35 so that the estimated encoding time becomes the time 
usable for the encoding of a picture determined in 
accordance with a frame rate or shorter (corresponding 
to claim 79). 

[0331 ] In the case of the second emtDodiment. it is also 
40 possible to add a two-byte region immediately after the 
start code shown in Figure 2 when the data string is 
generated in step 808 and add the binary notation of a 
code length to the region. 

[0332] Moreover, in the case of the fourth embodi- 
45 ment. it is also possible to extract a code length from tiie 
two-byte region when multiplexed data is input in step 
902 and use the code transmission time obtained from 
the code length and the code transmission rate for the 
execution frequency calculation in step 905 (tiie execu- 
50 tion frequency of halfjpixel predfotion is decreased so as 
not to exceed the code transmission time). This corre- 
sponds to claims 81 and 83. 

[0333] Furthermore, in the case of tiie first embodi- 
ment, it is also possible to add a two-bytid region imme- 
55 diately after the start code shown in Figure 2 when a 
data string is generated In step 2104 and add the binary 
notation of a code length to the region. 
[0334] Furthermore, in the case of tiie third embodi- 
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ment it Is also possible to extract a code length from the 
two-byte region when muttiplexed data is input in step 
301 and i^e a code transmission time obtained from the 
code length and the code transmission rate for the exe- 
cution frec^ency calculation in step 304 (the execution 
frequency of halfpixel prediction is decreased so as not 
to exceed the code transmission time). This corre- 
sponds to claims 82 and 84. 

[0335] Furthermore, in the case of the fourth emtxxji- 
ment. an actual execution frequency of halfpixel predic- 
tion is recorded immediately after step 909 to calculate 
a maximum value. When the maximum value is equal to 
or less than a small-enough value (e.g. 2 or 3), it is also 
possiUe to generate a data string (data string compris- 
ing a specific bit pattern) showing that halfpixel predic- 
tion IS not used and transrrut the generated data string. 
Furthermore, in the case of the second emtxxliment, it 
is confirmed whether the data string is received irrunedl- 
ately after step 808 and when the data string showing 
that halfpixel prediction is not used is received, it is also 
possiksle to make movement compensation processing 
always serve as fullpixet prediction in step 808. This cor- 
responds to claims 93 and 91. 

[0336] Furthermore, the above concept can be 
applied to cases other than movement compensation. 
For example, it is possit^e to reduce the DOT calcula- 
tion time by using no high-frequency component for 
OCT calculation. That is. in the case of a receiving 
method, when the rate of the IDCT-calculation execution 
time to the entire execution time exceeds a certain 
value, a data string showing that the rate exceeds a cer- 
tain value is transmitted to the transmitting side. When 
the transmitting side receives the data string. It Is also 
possil>le to calculate only low-frequency components 
through the DOT calculation and decrease all high-fre- 
quency components to zero. This corresponds to daim 
89. 

[0337] Furthermore, though the embodiment is 
described above by using a picture, it is possible to 
apply each of the above methods to audio instead of 
video. This corresporxJs to claims 85 and 87. 
[0338] Furthermore, in the case of the third embodi- 
ment, an actual execution frequency of haHjpixel predic- 
tion is recorded in st^ 3034 to calculate a maximum 
execution frequency. Then, when the maximum value is 
a small-enough value or less (e.g. 2 or 3), it Is possible 
to generate and transmit a data string showing that half- 
pixel prediction is not used (data string comprising a 
spedf ic bit pattern). Furthermore, in the case of the first 
endxxJiment, when receiving a data string showing that 
halfpixel prediction is not used, it is possible to make the 
movement compensation processing in step 1022 
always serve as fullpixel prediction. This corresponds to 
claims 94 arxJ 92. 

[0339] Furthermore, the above concept can t>e 
applied to cases other than movement compensation. 
For example, by using no high-frequency component for 
DOT calculation, it is possible to reduce the DCT calcu- 



lation processing time. That is, in the case of a receiving 
method, when the rate of IDCT-calculation execution 
time to the entire execution time exceeds a certEun 
value, a data string showing that the rate exceeds a cer- 

5 tain value is transmitted to the transmitting side. 

[0340] When the transmitting side receives the data 
string, it is possiUe to calculate only low-frequency 
components through the DCT calculation and reduce all 
high-frequency conponents to zera This oonresponds 

10 to claim 90. 

[0341] Furthermore, though the embodiment is 
described above by using a picture, it is also possible to 
apply the above method to audio instead of picture. This 
corresponds to claims 86 and 88. 

15 [0342] As descrS>ed above, according to claims 68 
and 74 (e.g. first and third embodiments), the execution 
time of decoding is estimated in accordance with the 
estimated execution time of each element and. when 
the estimated decoding execution time may exceed the 

20 time (designated time) required to receive the data for 
one sheet, halfpixel prediction having a long execution 
time is replaced with fullpixel prediction. Thereby, it is 
possit)le to prevent the execution time from exceeding 
the designated time and solve the problem (CI). 

25 [0343] Furthermore, according to claims 75 and 77 
(e.g. fifth and seventh embodiments), the estimated 
execution time of each element is transmitted to the 
transmitting side, the execution time of decoding is esti- 
mated at the transmitting side, and halfpixel prediction 

30 having a long execution time is replaced with fullpixel 
prediction so that the estimated decoding time does not 
exceed the time (designated time) probably required to 
receive tiie data for one sheet. Thereby, the information 
for halfpixel prediction in the sent encoded information 

35 is not disused and it is possible to prevent the execution 
time from exceeding the designated time and solve the 
problem (C2). 

[0344] Furthermore, according to claim 79 (e.g. ninth 
emtxxjiment). it Is possible to solve the problem (C3) by 

40 estimating the execution time of each processing, more- 
over estimating the execution time required for encoding 
in accordance with the estimated execution time, and 
deciding an executing frequency so that the estimated 
encoding time t^ecomes equal to or less than the time 

45 usable for encoding of a picture decided in accordance 
with a frame rate. 

[0345] Thus, the present invention makes it possit)le 
to realize a function (CGD: Computational Graceful 
Degradation) for slowly degrading quality even if a cal- 
50 culated load increases and thereby, a very large advan- 
tage can be obtained. 

[0346] Moreover, it is possit^le to perform operations 
same as described above by a computer by using a 
recording medium such as a magnetic recording 
55 medium or optical recording medium in which a pro- 
gram for making tiie computer execute all or part (or 
operations of each means) of the each steps (or each 
means) described in any one of the above^iescribed 
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embodiments. 
Industrial Applicability 

[0347] As described above, the present invention s 
makes it possible to change information frames corre- 
sponcfingly to the situation, purpose, or transmission 
line by dynamically deciding the frames off data control 
information, transmission control information, and con- 
trol information used for transmitting and receiving ter- io 
minats. Moreover, it is easy to handle a plifl-ality of video 
streams or a plurality of audio streams and mainly 
reproducing an important scene cut synchronously with 
audio by reflecting the intention of an editor. Further- 
more, it is possible to prevent an execution time from is 
exceeding a designated time by estimating the execu- 
tion time of decoding in accordance with the execution 
time of each estimated element and replacing halfpixel 
prediction having a long execution time with fullpixel 
prediction when the estimated decoding execution time 20 
may exceed the time (designated time) required to 
receive the data for one sheet 

Claims 

25. 

1. An audio-video transmitting apparatus comprising 
transmitting means for transmitting the content con- 
cerned with a transmitting method and/or the struc- 
ture ^of data to be transmitted or an identifier 
showing the content as transmission format intor- 30 
mation through a transmission line same as that of 
the data to be transmitted or a transmission line dif- 
ferent from the data transmission line; wherein 

said data to be transmitted is video data and/or 3S 
audio data. 

2. The audio-video transmitting apparatus according , 
to daim 1 . wherein said transmission format infor- 
mation is included in at least one of data control 40 
information added to said data to control said data, 
transmission control information added to said data 

to transmit said data, and information for controlling 
the processing of the terminal side. 

45 

3. The audio-video transmitting apparatus according 
to claim 2. wherein at least one of said data control 
information, transmission control information, and 
Information for controlling the processing of said 
terminal side in dynamically changed. so 

4. The audio-video transmitting apparatus according 
to claim 3. wherein 

said data is divided into a plurality of packets, ss 
and 

said data control information or said transmis- 
sion control information is added not only to the 



head packet of said divided packets but also to 
a middle packet of them. 

5. The audio-video transmitting apparatus according 
to daim 1 . wherein an identifier showing whether to 
use timing information concerned with said data as 
informatfon showing the reproducing time of said 
data IS included in said transmission format infor- 
mation. 

6. The audio-video transmitting apparatus according 
to daim 1, wherein said transmission format infor- 
matfon is the structural information of said data and 
a signal which is output from a receiving apparatus 
receiving the transmitted structural information of 
said data and which can be received is oonfirmed 

. and thereafter, said transmitting means transmits 
corresponding data to said receiving apparatus. 

7. The audio-video transmitting apparatus according 
to claim 1, wherein safo transmission format infor- 
mation indude (1) an identifier for identifying a pro- 
gram or data to be used by a receiving apparatus 
later and (2) at least one of a flag, counter, and 
timer as information for knowing the point of time in 
which said program or data is used or the term of 
validity for using said program or data. 

8. The audio-video transmitting apparatus according 
to daim 7, wherein said point of time in which said 
program or data is used is transmitted as transmis- 
sion control information by using a transmission 
serial number for identifying a transmissfon 
sequence or as information to be transmitted by a 
packet different from that of data to control terminal- 
side processing. 

9. The audio-video transmitting apparatus according 
to claim 2 or 3. wherein 

storing means for storing a plurcUlty of contents 
concerned witii said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are included, 
and 

said identifier is induded in at least one of said 
data control information, transmission control 
information, and information for controlling ter- 
minal-side processing as said transmission for- 
mat information. 

10. The audio-video transmitting apparatus according 
to daim 2 or 3, wherein storing means for storing a 
plurality of contents concerned with said transmit- 
ting method and/or said structure of data to be 
transmitted are included, and 

said contents are included in at least one of 
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said data control information, transmission 
- control information, and information for control- 
ling terminal-side processing as said transmis- 
sion format information. 

5 

11. The audio-video transmitting apparatus according 
to daim 1 . 2. or 3, wherein a default identifier show- 
ing whether to change the contents corx^emed with 
said transmitting method and/or structure of data to 

be transmitted is added. io 

12. The audio-video transmitting apparatus according 
to daim 9, 10. or 11 . wherein said identifier or said 
default identifier is added to a predetermined fixed- 
length region of information to t>e transmitted or is 
said predetermined position, 

13. An audio-video receiving apparatus comprising: 



arranging said infbrmation without mult^Diexing it 
t>efore.said data arxi/or control information or trans- 
mit saki multiplexing control information through a 
transmission line different from the transmission 
line for transmitting said data and/of control infor- 
mation. 

17. An aucfio-video receiving apparatus compristng: 

receiving means for receiving said multiplexing 
control information transmitted from the au(fio- 
video transmitting apparatus of daim 15 and 
said multiplexed data and/or control informa- 
tion; and 

sepiarating means for separating isaid multi- 
plexed . data and/or control information in 
accordance with said multiplexing control infor- 
mation. 



receiving means for receiving said transmis- 20 ia An aucfio-vkl€p receiving apparatus comprisirig: 
sion format information transmitted from the 
audio-video transmitting apparatus of any one 
of daims 1 to 12; and 

transmitted-information interpreting means for 
interpreting said received transmission-format ss 
information. 



14. The audio-vkfeo receiving apparatus according to 
daim 13, wfierein 
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Storing means for storing a plurality of contents 
corKerned with said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are induded. 
and 35 
the contents stored in said storing means are 
used to interpret said treuismission format infor- 
mationf 

1 5. An audio-video transmitting apparatus conprising: 40 

information multiplexing means for controlling 
start and erxJ of multiplexing tiie information for 
a plurality of logical transmission lines for trans- 
mitting data and/or control infbrmation is 45 
induded; wherein. 

not only said data and/or control information 
multiplexed by said information multiplexing 
means but also control contents concerned 
with start and end of said multiplexing by said so 

II llX/i IIK&UVt • I I IV»lbl|rfl%«'Vlt 1^ I ttWVkl «M Ml € MVM IWl ■ IIMW^* 

as multiplexing control information, and 

said data includes video data and/or audio 

data. 



main looking-listening means for looking at and 
listening to a broadcast program; and 
auxili£U7 looking-listening means for cydically 
detecting the state of. a broadcast program 
otiier than the broadcast program looked and 
listened through said main looking-listening 
means: wherein 

said detection is performed so that a program 
and/or data necessary when said broadcast 
program looked and listened through said main 
looking-listening means is switched to otiier 
broadcast program can be smoottily proc- 
essed, and 

said data indudes video data and/or audio 
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16. The audio-video transmitting apparatus according 
to claim 15, wherein it is possible to select whether 
to transmit said multiplexing control information by 



19. The audio-vkieo transmitting apparatus according 
to claim 1 , wherein priority values can be changed 
in accordance with the situation by transmitting the 
offset value of information showing the priority for 
processing of said data. 

20. An audio-video receiving apparatus comprising: 

receiving means for receiving encoded infor- 
mation to which the information concerned with 
the priority for processing under an overfoad 
state Is previously added; and 
priority dedding means for deoding a threshold 

eAr\/inn ac a rriforinn #nr calM^nn vuihothor tn 
....g ^ . ^-..5, — 

process an object in said information received 
by said receiving means; wherein 
the timing for outputting saki received informa- 
tion is compared with the elapsed time after 
start of processing or the timing for decoding 
said received information is compared with the 
elapsed time after start of processing to 
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change said threshold in accoidance with the 
. comparison result, and 
video data and/or audio data are or is included 
as said encoding object. 

5 

21. The audio-video receiving apparatus according to 
claim 20. wherein 

retransmission-request-priority deciding 
means for deciding a threshold serving as a cri- io 
terion for selecting whether to request retrans- 
mission of some of said information not 
received because it is lost under transmissfon 
when it is necessary to retransmit said informa- 
tion is included, and is 
said decided threshold is decided in accord- 
ance with at least one of the priority controlled 
by said priority deciding means, retransmission 
frequency, lost factor of information, insertion 
Interval between in-frame-encoded frames. 20 
arKi grading of priority. 

22. An audio-video transmitting apparatus comprising: 

retransmission-priority deciding means for 25 
deciding a threshold serving as a criterion for 
selecting whether to request retransmission of 
. some of said information not received because 
it is lost urxjer transmission when retransmis- 
sion of said unreceived information is 30 
requested is included, wherein 
said decided threshold is decided in accord- 
ance with at least one of the priority controlled 
by the priority deciding means of said audio- 
video receiving apparatus of daim 20. retrans- 3S 
mission frequency, lost foctor of information, 
insertion interval between in-frame-encoded 
frames, and grading of priority. 

23. An audio-video transmitting apparatus for transmit- 40 
ting said encoded information by using the priority 
added to said encoded information and thereby 
thinning it when (1) an actual transfer rate exceeds 
the target transfer rate of information for a video or 
audio or (2) it is decided that writing of said 45 
encoded information into a transmitting t)uffer is 
delayed as the result of comparing the elapsed time 
after start of transmission with a period to be 
decoded or output added to said encoded informa- 
tion, so 

24. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video; (2) an inter-time-series- ss 
data priority showing the priority of the 
processing between said time-series-data val- 
ues, and <3) a plurality of in-time-series-data 



priorities for dividing said time-series data 
value to show the processing priority between 
divided data values: and 
performing processing by using said inter-time- 
series-data priority and said in-time-series- 
data priority together when pluralities of said 
time-series-data values are simjltaneously 
present. 

25. A data processing apparatus comprising: 

receiving means for receiving a data series 
including (1) time-series data for audfo or 
video. (2) an inter-time-series-data priority 
showing the priority of the processing between 
said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said 
time-series data value to show the processing 
priority k>etween divided data values: and 
data processing means for performing 
processing by using said inter-time-series-data 
priority and said in-time-series-data priority 
tog^her when pluralities of said time-series- 
data values are simultaneously present 

26. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the 
processing between said time^eries-data val- 
ues, and (3) a plurality of In-time-series-data 
priorities for dividing said time-series data 
value to show the processing priority between 
divided data values: and 
distributing throughput to each of said time- 
series-data values in accordance with said 
inter-time-series-data priority and moreover, 
adaptively deteriorating the processing quality 
of the divided data in said time-series data in 
accordance with said in-time-series-data prior- 
ity so that each of said time-series-data values 
is kept within said distritxjted throughput. 

27. A data processing apparatus conrprising: 

receiving means for receiving a data series 
including (1) time-series data for audio or 
video. (2) an inter-time-series-data priority 
showing the priority of the processing between 
said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said 
time-series data value to show the processing 
priority between divided data values; and 
data processing means for distributing through- 
put to each of said time-series-data values in 
accordance with said inter-time-series-data pri- 
ority and moreover, adaptively deteriorating tiie 
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processing quality of the divided data in said 
time-series data in accordance with said in- 
time-series-data priori^ so that each of said 
time-series-data vaJues is kept within said dis- 
tributed throughput. 

28. A data processing method characterized by. when 
an in-time-series*data priority for a video is added 
every frame of said video and said video for each 
frame is divided Into a plurality of packets. 



34. A data processing method comprising the steps of: 

inputting a data series fnoluding time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- 
series-data priority as the value of a relative or 
absolute priority. 



10 



adding said in-time-series-data priority only to 
the header portion of a packet for transmitting 
tiie head portion of a frame of said video 
acoessft)le as independent information. is 

29. A data processing apparatus characterized by. 
when an in-time-series-data priority for a video is 
added every frame of said video and said video for 
each frame is divided into a plurality of packets. 20 

adding said in-time-series<lata priority only to 
the header portion of a packet for transmitting 
the head portion of a frame of said video 
accessSsle as independent information. 25 

30. The data processing metiiod acciording to any one 
of claims 24, 26. and 28, wherein said in-time- 
series-data priority is described in the header of a 
packet to perform prknity processing. 30 

31. The data processing apparatus according to any 
one of claims 25, 27. and 29. wherein said in-time- 
series-data priority is described in the header of a 
packet to pcNlbrm priority processing. as 

32. The data processing method according to any one 
of claims 24. 26. and 28. wherein tine range of a 
value capable of expressing said in-time-series- 
data priority is made variable to peribrm priority 40 
processing. 

33. the data processing apparatus according to any 
one of claims 25. 27. and 29. wherein the range of 

a value capable of expressing said in-time-series- 4s 
data prk>rity is made variable to perform priority 
processing. 



so 



ss 



35. A data processing apparatus characterized by: 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- 
series-data priority as the value of a relative or 
absolute priority. 

36. A data processing metiiod comprising tiie steps of: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality of in-time-series- 
data priorities showing the processing priority 
between said classified data values; and 
processing prforities by using saki in-time- 
series-data priority as the value of a relative or 
at)solute priority. 

37. A data processing apparatus characterized by: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality off in-time-series- 
data priorities showing the processing priority 
between sakJ classified data values; and 
processing priorities by using said in-time- 
series-data priority as the value of a relative or 
absolute priority. 

3& A data processing method comprising the steps of: 

classifying time-series data values for audio or 
video; 

encoding said classified data values; 
inputting a data series descrOding an in-time- 
series-data priority serving as the value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as the 
value of a relative priority in the header portion 
of a packet constituted with saki encoded infor- 
mation; and 
processing priorities. 

39. A data processing apparatus characterized by: 

w<«Miijriii^ uiiio-oeiieo uaict vctiutss lur aUQIO Or 

video; 

encoding said classified data values; 
inputting a data series describing an in-time- 
series-data priority serving as the value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as tiie 
value of a relative priority in the header portion 
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of a packet constituted with said encoded infor- 
mation; and 
processing priorities. 

40. A data processing method comprising the steps of: s 

inputting a data series including time-series 
data for audio or video and an inter-time- 
seriesKlata priority showing the processing pri- 
ority between time series data values; and io 
processing priorities fc>y relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

41 . A data processing apparatus characterized by: is 

Inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between time series data values: and 20 
processing priorities t>y relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

42. The data processing method according to claim 34 25 
or 36. wherein 

said priority processing is peribrmed (1) by 
using said inter-time-series-data priority as the 
value of a relative priority when accumulating 30 
and using said inter-time-series-data priority 
and (2) by using said inter-time-series-data pri- 
ority as the value of an absolute priority when 
transmitting said data. 

35 

43. The data processing apparatus according to claim 
35 or 37, wherein 

said priority processing is peribrmed (1) by 
expressing said inter-time-series-data priority 40 
as the value of a relative priority when accumu- 
lating and using said inter-time-series-data pri- 
ority and (2) by expressing said inter-time- 
series-data priority as the value of an absolute 
priority when transmitting said inter-time- 4S 
series-data priority. 

44. The data processing method according to claim 34 
or 36, wherein 

50 

an identifier classifies whether to express the 
value of said priority as a relative value or an 
absolute value. 

45. The data processing apparatus according to claim 55 
35 or 37. wherein 

an identifier classifies whether to express the 



value of said priority as a relative value or an 
absolute value. 

46. A data processing method comprising the steps of: 

when one time-series data includes a plurality 
of sub-time-series data values/describing the 
relation between said sub-time-series data val- 
ues and thereby defining a method for process- 
ing said sub-timerseries data to perform priority 
processing. 

47. A data processing apparatus characterized by, 
when one time-series data includes a plurality of 
sub-time-series data values, describing the relation 
between said sut>-time-series data values and 
thereby defining a method for processing said sub- 
time-series data to perform priority processing. 

48. The data processing method according to any one 
of claims 34. 36. and 46, wherein a packet-oonsti- 
tuting method is decided in accordance with any 
one of said inter-time-series-data priority, in-time- 
series-data priority, and relational description 
between sakJ time-series data values. 

49. The data processing apparatus according to any 
one of claims 35. 37. and 47, wherein a packet-con- 
stituting method is decided in accordance with any 
one of said inter-time-sertes-data priority, in-time- 
series-data priority, and relational description 
t>etween said time-series data valuesl 

50. A data processing method characterized by relating 
the sliced structure of a video to the structure of a 
packet and thereby, making a re-sync marker for 
resynchronization unnecessary. 

51. A data processing apparatus characterized by 
relating tiie sliced structure of a video to the struc- 
ture of a packet and thereby, making a re-sync 
marl^r for resynchronization unnecessary. 

52. A data processing apparatus characterized by 
transmitting a method for relating time-series data 
for audio or video to a packet together with control 
information or said time-series data arxl thereby, 
defining relating of said time-series data to saki 
pactet. 

53. The data processing method according to claim 48, 
wherein high en^or protection is applied to a packet 
including the information for said high in-time- 
series-data priority or inter-time-series-data priority. 

54. The data processing apparatus according to daim 
49, wherein high error protection is applied to a 
pad^t including the information for said high in- 
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time-series-data priority or Inter-time-series-data 
priority. 

55. The data processing method according to daim 34 
or 36. wherein 

a priority added to a packet is used as a padcet 
priority, and 

said priority processing is performed by relating 
at least either of the values of said in-time- 
series<lata priority and said Inter-time-serieS' 
data priority to said packet priority. 

56- The data processing apparatus according to daim 
35 or 37. wherein 

a priority added to a packet is used as a pactet 
priority, and 

said priority processing is performed by relating 
at least either of the values of said in-time- 
series-data priority and said inter-time-series- 
data priority to said packet priority 

57. The data processing method according to claim 34 
or 36. wherein 

said priority processing Is performed by assign- 
ing a value tower tiian a character or control 
information to said time-series data as the 
value of said in-time-series-data priority or said 
inter-time-series-data priority 

58. The data processing apparatus according to daim 
35 or 37. wherein said priority processing is per- 
formed by assigning a value lower than a character 
or control information to said time-series data as 
the value of said in-time-series-data priority or said 
inter-time-series-data priority 

59. A data processing method comprising the steps of: 

successively inputting dassified time-series 
data and its priority information; and 

(1 ) when the information for said classified 
time-series data is damaged, performing 
retransmission request processing in order 
to request retransnrtission of sakf damaged 
data and (2) when said classified time- 
series data is continuously or frequentiy 
lost, applying said retransmission request 
processing only to high-priority data. 

60. A data processing apparatus characterized by. suc- 
cessively inputting dassified time-series data and 
its priority information; and 

(1) when the Infbrrtiation for said dassified 



time-series data is damaged, performing 
retransmission request processing in order to 
request retransmission of said damaged data 
and. (2) when said classified time-series data 
5 is continuously or frequentiy lost applying sakl 

retransmission request processing only to 
high-priority data. 

61. A data processing metiiod com|»ising the step of: 

10 . 

successively inputting dassified time-series 
data and its priority information; and 
preferentially tiansmitting said high-priority 
data in accordance with the amount of saki 
IS dassified time-series data to be transmitted. 

62. A data processing apparatus characterized by: 

successively Inputting dassified time-series 
20 data and its priority information; and 

preferentially transmitting said high-priority 
data in accordance with the amount of sak:l 
dassified time-series data to be transmitted, 

25 63. A waveform data transmitting metiiod conprising 
the steps of: 

(a) dividing a plurality of decoding units consti- 
tuting the v»«vef6rm-data decoding process 
30 into a plurality of groups in accordance with tiie 

significance for maintaining quality and count- 
ing the execution frequency of an encoding unit 
corresponding to the decoding unit belonging 
to each group; 

35 (b) receiving said counted result and transform- 

ing said result into a data string when encoding 
of waveform data for a predetermined period is 
coirpleted; and 

(c) outputting a code which is a waveform-data 
<o encoding result and saki data string and trans- 

mitting the execution fi^equency of each 
processing unit every a plurality of; groups to 
the receiving apparatus. 

45 64. A waveforrn data ti-ansmitting apparatus compris- 
ing: 

(a) counting means for divicfing a plurality of 
decoding units constituting the waveform-data 

so decoding process into a plurality of groups in 

accordance with the signiffoance for maintain- 
ing quality and counting the execution fre- 
quency of an encoding unit corresponding to 
the decoding unit belonging to each group; 

ss (b) transforming means for receiving said 

counted result and transforming said result into 
a data string when encoding of waveform data 
for a predetermined period is completed; and 
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(c) transmitting means for outputting a code 
which is a wavefbrnfvdata encoding result and 
sard data string; wherein 
the execution frequency of each processing 
unit is transmitted to the receiving apparatus 
every a plurality of groups. 

65. The waveform data transmitting method according 
to daim 63. wherein 

pluralities of decoding units constituting a plu* 
rality of waveform-data decoding processes 
are divided into at least one indispensak)le 
processing or more and at least one dispense- 
ble processing or more (when this processing 
is on^ed. waveforms are deteriorated but 
waveforms can be decoded), the execution fre- 
quency of said indispensable processing arKi 
that of dispensable processing are counted, 
and the execution frequencies of said indispen- 
sable and dtspensat>le processings for each 
processing unit are transmitted to said receiv- 
ing apparatus. 

66. The waveform data transmitting apparatus accord- 
ing to claim 64, wherein counting means for dividing 
a plurality of decoding units constituting a plurality 
of waveform-data decoding processes into at least 
one indispensable processing or more and at least 
one dispensable processing or more (when this 
processing is omitted, waveforms are deteriorated 
but waveforms can be decoded) and counting the 
execution frequency of said indispensable process- 
ing and that of dispensable processing Is included 
and the execution frequencies of said indispensa- 
t)le and dispensable processings for each process- 
ing unit are transmitted to said receiving apparatus. 

67. The video waveform data transmitting method 
according to daim 63, wherein a video is input as 
said waveform data. 

68. The video waveform data transmitting apparatus 
according to claim 64. wherein a video is Input as 
said waveform data. 

69. A waveform data receiving method comprising the 
steps of: 

(a) receiving a data string including the code of 
waveform data and the execution frequency of 
each decoding unit grouped in accordance with 
tiie significance for maintaining the quality of 
the waveform data decoded from said code 
and outputting said code and said execution 
frequency: 

(b) estimating the execution time of each group 
in accordance with the processing time until 



obtaining a waveform after decoding said code 
and each of said execution frequencies 
obtained from said data string; and 
(c) estimating ttie processing time required to 

5 decode a waveform by using the execution fre- 

quency and said execution time, calculating the 
reduced number of execution frequencies of 
groups in which said processing time does not 
exceed the time required to receive said code 

10 or the time from start of receiving said code up 

to start of receiving the next code (this is 
referred to as designated time) |n accordance 
witii each execution time output by said receiv- 
ing means and each execution time output by 

IS said estimating means, estimating the time 

required for decoding, and redudng the execu- 
tion frequency of each group so as to complete 
decoding within said designated time. 

20 70. A waveform data receiving apparatus comprising: 

(a) receiving means for receiving a data string 
induding the code of waveform data and the 
execution frequency of each decoding unit 

25 grouped in accordance with the significance for 

maintaining the quality of the waveform data 
decoded from said code and outputting said 
code and said execution frequency; 

(b) estimating means for estimating the execu- 
30 tion time of each group in accordance with the 

processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequences obtained from said data 
string: and 

35 (c) frequency reducing means for estimating 

the processing time required to decode a wave- 
form by using said execution frequency and 
said execution time, calculating the reduced 
number of execution frequencies of the groups 

40 in which said processing time does not exceed 

the time required to receive said code or the 
time from start of receiving said code up to 
start of receiving the next code (this is referred 
to as designated time) in accordance with each 

45 execution time output by said receiving means 

arid each execution time output by said esti- 
mating means; wherein the time required for 
decoding is estimated and the execution fre- 
quency of each group is reduced so as to com- 

50 plete decoding within said designated time. 

71. A waveform data receiving method comprising tiie 
steps of: 

55 (a) receiving a data string including the code of 

waveform data and the execution frequencies 
of indispensat>le and dispensat)le processings 
for decoding and outputting said code and said 
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execution frequencies; 

(b) estimating tiie execution frequencies of said 
indispensable and dispensable processings in 
accordance with the processing time until 
obtaining a waveform after decocfing said code s 
and each of said execution frequencies 
obtained from said data string; and 

(c) estimating the processing time required to 
decode a waveform by using said execution 
frequency and said execution time, calculating 10 
the reduced number of execution frequencies 

of said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or ttie time from 
start of receiving said code up to start of receiv- is 
ing the next code (this is referred to as desig- 
nated time) in accordance with each execution 
frequency output by said receiving means and 
each execution time output by said estimating 
means, estimating the time required for decod- 20 
ing in accordance with each estimated execu> 
tion time, and reducing the execution frequency 
of said dispensable processing so as to com- 
plete decoding within said designated time. 

72. A waveform data receiving apparatus comprising: 

(a) receiving means for receiving a data string 
including the code of waveform data and the 
execution frequencies of indispensable and 30 
dispensable processings for decoding and out- 
putting said code and said execution frequen- 
cies; 

(b) estimating means for estimating the execu- 
tion frequencies of said indispensable and dis- 3S 
pensabie processings in accordance witii the 
processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequencies obtained from said data 
string; and 40 

(c) frequency reducing means for estimating 
the processing time required to decode a wave- 
form by using said execution frequency and 
said execution time and calculating the 
reduced number of execution frequendes of 45 
said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or tiie time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- so 

natArl tirYYa^ !n a^«A^r^«km^A t^.;*!* ^■^^^ ^'.^^ ■ 
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frequency output by said receiving means cuid 
each execution time output by said estimating 
means; wherein 

the time reqiired for decoding is estimated in ss 
accordance with each estimated execution 
time and the execution frequency of said dis- 
pensable processing is reduced so as to com- 



plete decoding within said designated time. 

73. The video waveform data receiving method accord- 
ing to daim 69. wherein a video is output as said 
waveform data. 

74. The video waveform data receiving apparatus 
according to daim 70, wherein a video is output as 
Scud waveform data. 

75. The video waveform data receiving metiiod accord- 
ing to daim 69, wherein (d) tiie execution time of 
each group obtained through estimation is output 

76. The video waveform data receiving apparatus 
according to daim 70. wherein (d) the execution 
time of each group ot>tained by estimating means is 
output. 

77. The waveform data transmitting method accoiding 
to daim 63. wherein. 

(d) a data string including the execution time of 
each group is input, and 

(e) the execution frequency of each group is 
calculated in accordance with each execution 
time of said receiving means in order to com- 
plete decoding within the time required to 
transmit a code lengtti dedded by the designa- 
tion by a rate controller or tiie like. 

78. The waveform data transmitting apparatus compris- 
ing: 

(d) receiving means for inputting a data string 
constituted witii the execution time of each 
group; and 

(e) deciding means for calculating the execu- 
tion frequency of each group in accordance 
with each execution time of said receiving 
means in order to conplete decoding within the 
time required to transmit a code decided by the 
designation by a rate controller or the lite. 

79. The video waveform data transmitting mettiod 
according to daim 67. wherein 

(d) the execution time of each group Is esti- 
mated in accordance with the processing time 
reqinred to encode a video and said each exe- 
cution irequenoy; and 

(e) the processing time required to encode a 
video is estimated by using said execution time 
and the execution frequency of each group is 
calculated in which said processing time does 
not exceed the time usat^ie to process a sheet 
of video determined in accordance with a 
frame rate given as tiie designation by a user. 
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80. A video waveform data transmitting apparatus 
according to claim 68. wherein 

(d) estimating means for estimating the execu- 
tion time of each group in accordance with the s 
processing time required to encode a video 
and each execution time output by counting 
means; and 

(e) deciding means for estimating the process- 
ing time required to encode a video by using w 
said iexecution time and calculating the execu- 
tion frequency of each group in which said 
processing time does not exceed the time usa- 
ble to process a sheet of video determined in 
accordance with a frame rate given as the des- is 
ignation tjy a user. 

81. The video waveform data transmitting method 
according to daim 63. wherein said counting result 
and the length of a code corresponding to wave- 20 
form data for a predetermined period are received 
when generation of said code Is completed to trans- 
form the result and length into a data string. 

82. The video waveform data transmitting apparatus 25 
according to claim 63, wherein transforming means 

is included which receives said counting result of 
said counting means and the length of a code cor- 
responding to waveform data for a predetermined 
period when generation of said cede is completed so 
to transform the result and length into a data string. 

83. The waveform data receiving method according to 
claim 69, wherein a data string including a code 
corresponding to the waveform data for a predeter- 35 
mined period, the execution frequency of each 
decoding unit grouped in accordance with the sig- 
nificarx;e for maintaining the quality of the wave- 
form data decoded from said code, and the length 

of said code is received and said code, execution 40 
frequency, and code length are output to reduce the 
execution frequency of dispensable processing so 
that the time required for decoding does not exceed 
a code transmission time obtained from the length 
and transmission rate of said code. 4S 

84. The waveform data receiving apparatus according 
to daim 70, v/herein receiving means is included 
which receives a data string including a code corre- 
sponding to the waveform data for a predetermined so 
period, the execution frequency of each decoding 
unit grouped in accordance with the significance for 
maintaining the quality of the waveform data 
decoded from said code, and the length of said 
code and outputs said code, exeoition frequency, ss 
and code length to reduce the execution frequency 

of dispensable processing so that the time required 
for decoding does not exceed a code transmission 



time obtained from the lengtti and transmission rate 
of said code. 

85. A waveform data receiving method for receiving the 
code of waveform data and decoding and output- 
ting the waveform, comprising the steps of: 

(a) constituting a data string induding the des- 
ignation for selecting a processing unit having 
an execution time shorter than that of tiie 
encoding unit induded in said code every 
encoding unit corresponding to a processing 
unit constituting the decoding process so that 
the processing time required to decode a wave- 
form does not exceed the time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the 
next code (this is referred to as designated 
time); and 

(b) transmitting said data string to communi- 
cate to the transmitting side that a code for 
completing decoding within said designated 
time is transmitted. 

86. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting said waveform, comprising: 

(a) designated data constituting means for con- 
stituting a data string induding the designation 
for selecting a processing unit having an esxecu- 
tion time shorter than that of the encoding unit 
induded in said code every encoding unit cor- 
responding to a processing unit constituting the 
decoding process so that the processing time 
required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time); and 

(b) transmitting means for transmititing said 
data string; wherein 

it Is communicated to the ti^nsmittihg side that 
a code for completing decoding within said 
designated time is transmitted. 

87. A waveform data transmitting method for encoding 
a waveform and outputting said code, comprising 
the steps of: 

(a) receiving a data string including the desig- 
nation for a processing unit to be selected for 
each processing unit constituting the encoding 
process; and 

(b) extracting said designation from said data 
string, encoding a waveform by using the 
processing unit specified in accordance with 
said designation, and outputting a code. 



39 



77 



EP0905 976A1 



78 



88. A waveform data transmitting apparatus for encod- 
ing a waveform and outputting said code, compris- 
ing: 

(a) receiving means for receiving a data string s 
including the designation for a processing unit 

to be selected for each processing unit consti- 
tuting the encoding process: and 

(b) extracting means for extracting said desig- 
nation from said data string; wherein io 
a waveform is encoded by using the processing 
unit specified in accordance with said designa- 

. tion to output a code. 

89. A waveform data receiving method for receiving the is 
code of waveform data and decoding and output- 
ting a waveform, comprising the steps of: 

(a) counting the execution frequency of each 
processing unit constituting the waveform 20 
decoding process; 

(b) estimating the execution time for each 
processing unit in accordance with said execu- 
tion frequency and processing time required to 
decode a waveform; 2s 

(c) constituting a data string including the des- 
ignation for selecting a processing unit having 
an execution time shorter than that of the 
encoding unit included in said code every 
encoding unit corresponding to the processing 30 
unit constituting the decoding process so that 
the processing time required to decode a wave- 
form does not exceed the time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the as 
next code (this is referred to as designated 
time); and 

(d) transmitting said data string; wherein 

it is communicated to the transmitting method 
that a code for completing decoding within said 40 
designated time is transmitted. 

90. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting a waveform, conprising: 45 

(a) counting means for counting the execution 
frequency of each processing unit constituting 
the waveform decoding process; 

(b) estimating means for estimating the execu- so 
tion time for each processing unit In accord- 
ance with said execution frequency and 
processing time required to decode a wave- 
form; 

(c) designated-data constituting means for con- ss 
stitutihg a data string including the designation 

for selecting a processing unit having an execu- 
tion time shorter than that of the encoding unit 



included in said code every encoding unit cor- 
responding to the processing unit constituting 
the decoding process so that the processing 
time required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time); and 
(d) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing decoding within said 
designated time Is transmitted. 

91. A video waveform data receiving method for recav- 
ing the code of a video and decoding and outputting 
said video, comprising the steps of: 

(a) constituting a data sti-ing including tiie des- 
ignation for replacing the movement cornpen- 
sating method used to encode a video with the 
movement compensation processing having an 
execution time shorter tiian that of the move- 
ment compensation processing included in 
said code so that the processing time required 
to decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- 
nated time); and 

(b) transmitting said data string; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 

92. A video receiving apparatus for receiving the code 
of a video and decoding and outputting said video, 
comprising: 

(a) designated-data constituting means for 
constituting a data sti^ing including the designa- 
tion for replacing the movement compensating 
metiiod used to encode a video witii the move- 
ment compensation processing having an exe- 
cution time shorter tiian that of the movement 
compensation processing included in said 
code so that the processing time required to 
decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is refened to as desig- 
nated time); and 

(b) tiBnsmitting means for transmitting said 
data string ; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 
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93. A video transmitting method for encoding a video 
and oulputting said code, comprising tlie steps of: 

(a) receiving a data string including the desig- 
nation for the processing to be selected by s 
using the movement compensating processing 
constituting the decoding process; and 

(b) extracting said designation from said data 
string: wherein 

encoding of a video is executed by using the 10 
movement compensating processing specified 
in accordance witii said designation to output a 
code. 

94. A video transmitting apparatus for encoding a video is 
and outputting said code, comprising the steps of: 

(a) receiving means for receiving a data string 
including the desi^tion for the processing to 

be selected by using the movement cpmpen- 20 
sating processing constituting the decoding 

process; and 

(b) extracting means for extracting said desig- 
nation from said data string: wherein 

encoding of a video is executed by using the 25 
movement compensating processing specified 
in accordance with said designation to output a 
code. 

30 
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o Information showing start position capable of 
processing data or not 

• Flag for randcxn access (Random access flag)^ 
e.g. Intra- freune (I-picture) In the case of 
picture 

* Flag showing access unit (Access flag), 
e.g. Fr£une In the case of picture, GOB unit 



AL : Adaptation layer 
ES : Elementary stream 
PTS : Presentation -time •stamp 



Header 
Information 
of data 



Data (Picture or sound for each frame) 




• Information showing start position capable of 
processing data or not 

•Information showing data reproducing time (PTS) 

• Information showing data processing priority 
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oTStTransport streamCTransmission packet) 



TS 





AL 



ES 



Communication 
header 



AL 



ES 



Cocnmunication 
header 

"1— 

• Information showing start position capable of processing 
pieces of data or not 

• Identification number for showing data sequence(Sequence 
number) - 

• Time concerned with transmission of pieces of data 

o Handling time stamp and marker bit 



(a) 


Communication 
header 


AL 


ES 




t t 

Time stamp 1 
PTS or not (Additional) 




(b) 


Communication 
header 


AL 


ES 



T 



T 



Time stamp for 

communication 

header 



Time stamp PTS or not (Additional) 



(c) 



Communication 
header 

? 

MarkerBit 



AL 



ES 



Substituted by 
AL flag (Additional) 



(d) 



AL 



ES 




MarkerBit of 

communication 

header 



Communication 
header 

} 

MarkerBit 

(It is interpreted that random access flag 
and access flag are present in AL.) 
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i s . 5(a) 




! ES i 




Gommunl- 




ES 


ca1:lon 


AL 


header 







Communi- 






Communlr 




cation 


ES 


• • • 


cation 


ES 


header 






header 



i « . 



( b ) 




AL 



T r 

I ES I 

J '. L 




ES 



/ 



/ 



AL 



ES 



Communl- 








Communi- 






oatlon 


AL 


ES 


• • • • 


cation 


AL 


ES 


header 








header 
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H.223 or 
the like 




Data and 

control Information 



UDP/TCP/'RTP 




Intra- net and Inter -net 

or the like 
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i « . 



Broadcast program transmitting procedure 
<Broadcast type and communication type including return channel) 
Transmitting side Receiving side 

Transfer of data structure 



ACK/Reject 



(LGN O) : (*1) 



Transfer of corresponding data 
(From eacli port) : (*2) 



Are processing and 
reception possible? 
.Start decoding of 
data which can be 
decoded and display 
it. 



<Broadcast type (with no return channel)) 

Transmitting side Receiving side 

Transfer of program 
information and data structure 
(LCN O) : UDP(*3) 



Transfer of corresponding data 
(From each port) : UDP 



( * 1 ) Must be a system for detecting and retransmitting a packet 

loss like TCP. 
(*2) RTP/RTCP or TCP/IP 

(*3) Same data (picture or sound) or control information (broadcast 
program or data structure) is continuously repeatedly 
transmitted. A packet is detected and sequence is kept at a 
receiving terminal in accordance witha sequence number. (To be 
used in a local closed region. Traffic becomes too large.) 
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F i « . 1 O ( a ) 

Receiving terminal 



Control information 
or data 



Program or data 
to be required 



Flag, counter, 
or timer showing 
point of time to be 
required 



Main 

looking-listening 
section 



Auxiliary 
looking-Ustening 
section 



Storing 
section 



Output 
section 



JET i s . 1 O ( b ) 

Receiving terminal 



Control information 
or data 



Caption 
broadcast-program 
receiving section 









Main 

looking-listening 

section 




Output 
section 









Storing 
section 
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i IS • 11(a) 

<Hlerarchlcal image of object> 




<Transmlsslon Image of object> 
< 1 . Broadcast type > 




<2. Communication type> 

RTP/RTCP (Program ID of eactx logical 



channel is fixed. } 









Terminal 




— — — ' » 


Terminal 


A 


^ ; , 




B 









LCNO (control) 
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f i S 



1 Kb) 



—Capability excliange definitlonsCoriginal from H.245) 



TerminalCapabilitySet 



{ 



sequenceNumber 



::=SEQUENCE 
SequenceNumber, 



multiplexCapability 
capabiUtyTable 

capabiUtyDescriptors 

mpeg4CapabiUty 



MuLtiplexCapabilityOPTIONAL 
SET SIZEa.256) OF Capability 
TableEntryOPTIONAL, 
SET SIZE(1..256) OF Capability 
DescriptorOPTIONAL. 
MPEG4CapabilityOPTIONAL. 
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i « . 12 



— MPEG4 Capability definitions 



IV1PEG4Capability 
{ 

sequenceNumber 

NumberOfProcessObject 

{ 

l\/laxNumberOf Video 
iS/IaxNumberOf Sounds 
MaxNumberdflVlux 

I 

reconfigurationALCapability 

• « • 

MPEG4CapabilityAck 
sequenceNumber 

♦ • • 

MPEG4CapabiUtyReject 



sequenceNumber 

NumberOfProcessObject 

{ 

maxNumberOf Video 
maxNumberOf Sounds 
IVlaxNumberOfMux 
reconf i gurationALCapability 



::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

INTEGER(0..1023), 
INTEGER(0..1023). 
INTEGER(0..1023), 
BOOLEAN. 

::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

MaxNumberOf Video, 

iVIaxNumberOf Sounds 
maxNumberOf Mux, 

BOOLEAN. 
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i 



13(a) 



-^Group MUX definitions 



CreateGroupMux 



{ 



sequenceNumber 

GroupMuxID 
lanp or t Number 



} 

CreateGroupMuxAck 



sequenceNumber 



CreateGroupMuxReject 
{ 

sequenceNumber 
cause 



} 



::=SEQUENCE 

SequenceNumber, 

INTEGER(0..1023). 
LANPortNumber, 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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FT i « . 1 3 ( b ) 

DestoryGroupMux 

sequenceNumber 
GroupMuxID 

• • • 

DestoryGroupMuxAck 
sequenceNumber 

• • • 

DestoryGroupMuxReject 

^ sequenceNumber 
cause 
{ 

} 

• • • 

} 



::=SEQUENCE 

SequenceNumber, 
INTEGER(0..1023). 

::=SEQUENCE 
SequenceNumber. 

::=SEQUENCE 

SequenceNumber, 
CHOICE 



58 



EP 0 905 976 A1 



i « - 



13(c) 



PortNumberStructure 



::=SEQUENCE 



sequenceNumber SequenceNumber. 
lanPortNumber LANPortNumber. 
numberOfLogicalNumber rNTEGER(1..15). 
SEQUENCE SI2E0..15) OF PortStructureElement. 



PortStructureElement 
^ logicalPortNumber 

• • • 

f='ortNumberStructureAck 
sequenceNumber 

• • ♦ ■ 

PortNumberStructureReject 



{ 



sequenceNumber 

csdjse 

( 



::=SEQUENCE 
LogicalPortNumber, 

::=SEQUENCE 
SequenceNumber, 



::=SEQUENCE 

SequenceNumber, 
CHOICE 
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—Logical channel signalling definitions(original from H.245) 
— MPEG4 Object Create Operation(f or LANPortNumber) 



fowardLogicalChannelNumber LogicalChannelNumber, 

fowardLogicalChannelfParameters SEQUENCE 
{ 



h222LogicallChannelParameters H222LogicalChamelParameters, 
h223LogicalChannelParameters H223LogicalChannelParameters, 
v76LogicalChannelParameters v76LogicalChannelParameters, 
• • • » 

h2250LogicalChannelParanneters H2250LogicalChamelParameters, 
h223AnnexALogicalChannelParameters 
H223AnnexALogicalChannelParameters 
MPEG4LogicalChannelParameters MPEG4LogicaiChanelParafneters, 



OpenLogicalChannel 



::=SEQUENCE 



PortNumber 
dataType 

multiplexParameters 



INTEGER(0..65535XPTI0NAL, 

DataType, 

CHOICE 
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F- i 



1 S 



MPEG4LogicalChannelF^rameters ::=SEQUENCE 



-H.225BASE 
LANportNumber 
ProgramID 
ProgramName 



} 



BroadcastChannelProgram 



{ 



sequenceNumber 
numberOfChannelNumber 



INTEGER(0.. 65535), 
INTEGER(0..255). 

0CTETSTRING(SIZE(128)). 



::=SEQUENCE 

SequenceNumber, 
INTEGER(0,.1023). 



SEQUENCE SIZE(1..1023) OF MPEG4LogicalChamelParameters 



ChangeLogicalChannelAttribute 



{ 



sequenceNumber 

LanportNumber 

ProgramID 



} 

ChangeLogicalChannelAttributeAck 



{ 



sequenceNumber 



Cl^hgeLogicaiChannelAttributeReject 



sequenceNumber 

cause 

1 



::=SEQUENCE 

SequenceNumber 
LANPortNumber. 
INTEGER(0.-255), 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber. 
CHOICE 
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i « . 



1 6( a ) 



-MPEG4 Object Class definition 



MPEG4 Object Class definition 



:i=SEQUENCE 



{ 



] 



sequenceNumber SequenceNumber, 

ProgramlD INTEGER(0..255). 
NumberOfObjectsList INTEGER(0..1023). 
SEQUENCE SIZE(1..1023) OF ObjectStructureElement 



ObjectStructireBement 



{ 



SSRC 

LANPortNumber 



ScrambleFlag 
CGDOffset 
MediaType 



::=SEQUENCE 

INTEGER(0..16777215). 
INTEGER(102.4.5000), 

~f orRPTCVideo&Sound) 
BOOLEAN. 
INTEGER(0..255). 
INTEGER(0..255). 



IVIPEG4 Object Class def initionAck 



sequenceNumber 



MPEG4 Object Class def initionReject 



sequenceNumber 
cause 



{ 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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F i 



1- e( b ) 



—Adaptation Layer Reconfiguration Request definitions 



ALReconfiguration 



{ 



sequenceNumber 
R^domAccessFlagMaxBit 
PresentationTimeStampsMaxBit 
CGDPriorityMaxBIt 



::=CHOICE 

SequenceNumber. 
INTEGER(0...2), . 
INTEGER(0...32). 
INTEGER(0...8), 

— f orVideo and Sound 



—Adaptation Layer Reconfiguration Response definitions 



ALReconfigurationAck 
{ 

sequenceNumber 

• mm 

) 

ALReconfigurationReject 

sequenceNumber 

cause 

{ 



::=SEQUENCE 
SequenceNumber. 

::=SEQUENCE 

SequenceNumber, 
CHOICE 



<Relatlon between AL, ES. and RTP> 



AL 


1 ■■ r 

1 ES ; 

— 1 1 — 




TS 

RTF Header 


AL 


ES 




RTP Header 


ES 
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F' i. S 



1 'Z 



-Setup Program and Data Request definitions 



Setup Request 



{ 



:j=GHOICE 



sequenceNumber 



SSRC IMEGER(0..16777215)2'32, 



SequenceNumber. 



Logical Channel Number, 
setupitem 

{ 

executeProgramNumber 
dalaNumber 

exeeuteCommandNumber 
nofitycounter 



flag 
counter 
timer 



INTEGER(1024...5000). 
CHOICE 

INTEGER(0...255). 
INTEGER(0...255). 
INTEGER(0...255). 

CHOICE 

BOOLEAN 

INTEGER(0...255). 
INTEGER(0...255), 
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IT i s . IS 



—control and AL attribute definitions 



ControlALdefinition 
{ 

sequenceNumber 
AL 



i 



RandomAccessFlagUse 

PresentationTimeStampUse 

CGDPriorityUse 



::=CHOICE 



SequenceNumber; 
CHOICE 

BOOLEAN, 
BOOLEAN. 
BOOLEAN. 
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i s . 1 ©( a ) 

classES_header{ 

uint(4) headerlD; 
uint(24) bufferSizeES; 
uint(l) useTimeStamps; 



} 



u i n t ( 1 6 ) sequenceNumberMaxBit; 
u i n t ( 1 ) useHeaderExtension; 
if (useHeaderExtension)! 



} 

uint(3) reserved: 



uint(l) accessUintStartFlag; 

u i nt n ) randomAccessPointFlag; 

uintcO OCRsetFlag; 

u i nt(4) degradationPriorityMaxBit; 
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i 



X 9( b ) 



-Adaptation Layer PDU header configuration Request and Command definition 



AL configuration 



{ 



sequenceNumber 
defauUHeaderConfiguration 

headerlD 

MPEG4ALPDUHeaderConfig 



accessUintStartFlag 

randomAccessPolntFlag 

OCRsetFlag 

degradationPrlorityMaxBit 



::=SEQUENCE 

SequenceNumber. 
BOOLEAN. 
INTEGER(0..4). 
SEQUENCE 

BOOLEAN. 
BOOLEAN. 
BOOLEAN, 
INTEGER(0..4). 
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2 S 



Processing at receiving terminal under overloadCCommon to dynamic 
picture and sound) 

Thread for processing sound at system level is previously set it's 
processing priority to a value tiigher than that of thread for 
processing picture. 



Frame_skipped 
. =FLASE 



Set priority of 

frame to be 
disused to value 

larger than 
(CutOffPriority) 
"maxPriority". 



Step 101 
Value of resolution of 
priority to be added to 
frame, CGDof fset(Can be 
determined in accordance 
with receiving-terminal 
performance) is transmitted 
as control information. 

Step 103 



Step 102 




Step 104 



Frame_skipped=FLASE 



Frame_skipped 
=TRUE 



Deliver data to decoder. 

i 
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F- i 



2 T( a ) 
A031 



.4032 



/ 




/ 




Priority adding 








Priority deciding 






section 
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